Large-sensing-area under-display optical sensor

ABSTRACT

Optical sensing is provided with a large sensing area in a thin package. For example, embodiments can operate in context of an under-display optical fingerprint sensor integrated into an electronic device, such as a smartphone. Responsive to reflected probe light passing through a display module, a reflective structure is configured to redirect the reflected probe light onto a refractive structure, and the refractive structure is configured to converge the reflected probe light into an input aperture of an optical sensor for detection. Some embodiments operate in context of an enhancement panel having micro-prism structures that tend to blur the reflected probe light. In such context, embodiments are configured for off-axis detection to prefer light passing through only certain micro-prism faces, thereby mitigating blurring.

CROSS REFERENCE TO RELATED APPLICATIONS

This patent document claims the benefits and priority of, and is acontinuation-in-part application of, U.S. patent application Ser. No.15/708,088. filed on Sep. 18, 2017, which claims the benefits andpriorities of U.S. Provisional Patent Application No. 62/396,153, filedon Sep. 17, 2016; U.S. Provisional Patent Application No. 62/412,777,filed on Oct. 25, 2016; and U.S. Provisional Patent Application No.62/468,337, filed on Mar. 7, 2017.

U.S. patent application Ser. No. 15/708,038 claims the benefits andpriority of, and is a continuation in-part application of, U.S. patentapplication Ser. No. 15/616,856, filed on Jun. 7, 2017, which claims thebenefits and priorities of U.S. Provisional Patent Application No.62/347,073, filed on Jun. 7, 2016; U.S. Provisional Patent ApplicationNo. 62/363,832, filed on Jul. 18, 2016, and U.S. Provisional PatentApplication No. 62/363,823, filed on Jul. 18, 2016. In addition, U.S.patent application Ser. No. 15/616,856 claims the benefits and priorityof, and is a continuation-in-part application of, U.S. patentapplication Ser. No. 15/421,249, filed on Jan. 31, 2017, which claimsthe benefits and priorities of U.S. Provisional Patent Application No.62/289,328, filed on Jan. 31, 2016; U.S. Provisional Patent ApplicationNo. 62/330,833, filed on May 2, 2016, and U.S. Provisional PatentApplication No. 62/347,073, filed on Jun. 7, 2016. Furthermore, U.S.patent application Ser. No. 15/616,856 claims the benefits and priorityof, and is a continuation-in-part application of International PatentApplication No. PCT/US2016/038445, filed on Jun. 20, 2016, which claimsthe benefits and priority of U.S. Provisional Patent Application No.62/181,718, filed on Jun. 18, 2015.

U.S. patent application Ser. No. 15/708,038 also claims the benefits andpriority of, and is a continuation-in-part application of, U.S. patentapplication Ser. No. 15/421,249, filed on Jan. 31, 2017, which claimsthe benefits and priorities of U.S. Provisional Patent Application No.62/289,328, filed on Jan. 31, 2016, U.S. Provisional Patent ApplicationNo. 62/330,833, filed on May 2, 2016; and U.S. Provisional PatentApplication No. 62/347,073, filed on Jun. 7, 2016. In addition, U.S.patent application Ser. No. 15/421,249 claims the benefits and priorityof, and is a continuation-in-part application of. International PatentApplication No. PCT/US2016/038445, filed on Jun. 20, 2016, which claimsdie benefits and priority of U.S. Provisional Patent Application No.62/181,718, filed on Jun. 18, 2015.

U.S. patent application Ser. No. 15/708,058 also claims the benefits andpriority of, and is a continuation-in-part application of, InternationalPatent Application No. PCT/US2016/038445, filed on Jun. 20, 2016, whichclaims the benefits and priority of U.S. Provisional Patent ApplicationNo. 62/181,718, filed on Jun. 18, 2015.

U.S. patent application Ser. No. 15/708,088 also claims the benefits andpriority of, and is a continuation-in-part application of. InternationalPatent Application No. PCT/CN2016/104354, filed on Nov. 2, 2016, whichclaims the benefits and priority of U.S. Provisional Patent ApplicationNo. 62/249,832, filed on Nov. 2, 2015.

The entire contents of the before-mentioned patent applications areincorporated by reference as part of the disclosure of this document.

TECHNICAL FIELD

This disclosure relates to optical sensors, such as an under-screenoptical fingerprint sensors integrated with a display panel arrangementof mobile computing device and configured to provide a large sensingarea.

BACKGROUND

Various sensors can be implemented in electronic devices or systems toprovide certain desired functions. A sensor that enables userauthentication is one example of sensors to protect personal data andprevent unauthorized access n various devices and systems includingportable or mobile computing devices (e.g., laptops, tablets,smartphones), gaming systems, various databases, information systems orlarger computer-controlled systems.

User authentication on an electronic device or system can be carried outthrough one or multiple forms of biometric identifiers, which can beused alone or in addition to conventional password authenticationmethods. A popular form of biometric identifiers is a person'sfingerprint pattern. A fingerprint sensor can be built into theelectronic device to read a user's fingerprint pattern so that thedevice can only be unlocked by an authorized user of the device throughauthentication of the authorized user's fingerprint pattern. Anotherexample of sensors for electronic devices or systems is a biomedicalsensor that detects a biological property of a user, e.g., a property ofa user's blood, the heartbeat, in wearable devices like wrist banddevices or watches. In general, different sensors can be provided inelectronic devices to achieve different sensing operations andfunctions.

Fingerprints can be used to authenticate users for accessing electronicdevices, computer-controlled systems, electronic databases orinformation systems, either used as a stand-alone authentication methodor in combination with one or more other authentication methods such asa password authentication method. For example, electronic devicesincluding portable or mobile computing devices, such as laptops,tablets, smartphones, and gaming systems can employ user authenticationmechanisms to protect personal data and prevent unauthorized access. Inanother example, a computer or a computer-controlled device or systemfor an organization or enterprise should be secured to allow onlyauthorized personnel to access in order to protect the information orthe use of the device or system for the organization or enterprise. Theinformation stored in portable devices and computer-controlleddatabases, devices or systems, may be personal in nature, such aspersonal contacts or phonebook, personal photos, personal healthinformation or other personal information, or confidential informationfor proprietary use by an organization or enterprise, such as businessfinancial information, employee data, trade secrets and oilierproprietary information. If the security of the access to the electronicdevice or system is compromised, these data may be accessed by others,causing loss of privacy of individuals or loss of valuable confidentialinformation. Beyond security of information, securing access tocomputers and computer-controlled devices or systems also allowsafeguard the use of devices or systems that are controlled by computersor computer processors such as computer-controlled automobiles and othersystems such as ATMs.

Secured access to a device (e.g., a mobile device) or a system (e.g., anelectronic database and a computer-controlled system) can be achieved indifferent ways such as the use of user passwords. A password, however,may be easily to be spread or obtained and this nature of passwords canreduce the level of the security of passwords. Moreover, since a userneeds to remember a password in accessing password-protected electronicdevices or systems, in the event that the user forgets the password, theuser needs to undertake certain password recovery procedures to getauthenticated or otherwise to regain the access to the device or system.Such processes may be burdensome to users and have various practicallimitations and inconveniences. The personal fingerprint identificationcan be utilized to achieve the user authentication for enhancing thedata security while mitigating certain undesired effects associated withpasswords.

Electronic devices or systems, including portable or mobile computingdevices, may employ user authentication through one or multiple forms ofbiometric identifiers to protect personal or other confidential data andprevent unauthorized access. A biometric identifier can be used alone orin combination with a password authentication method to provide userauthentication. One form of biometric identifiers is a person'sfingerprint pattern. A fingerprint sensor can be built into anelectronic device or an information system to read a user's fingerprintpattern so that the device can only be unlocked by an authorized user ofthe device through authentication of the authorized user's fingerprintpattern.

SUMMARY

Embodiments provide optical sensing with a large sensing area in a thinpackage. For example, embodiments can operate in context of anunder-display optical fingerprint sensor integrated into an electronicdevice, such as a smartphone. Responsive to reflected probe lightpassing through a display module, a reflective structure is configuredto redirect the reflected probe light onto a refractive structure, andthe refractive structure is configured to converge the reflected probelight into an input aperture of an optical sensor for detection. Someembodiments operate in context of an enhancement panel havingmicro-prism structures that tend to blur the reflected probe light. Insuch context, embodiments are configured for off-axis detection toprefer light passing through only certain micro-prism faces, therebymitigating blurring.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, referred to herein and constituting a parthereof, illustrate embodiments of the disclosure. The drawings togetherwith the description serve to explain the principles of the invention.

FIG. 1 is a block diagram of an example of a system with a fingerprintsensing module which can be implemented to include an opticalfingerprint sensor according to some embodiments.

FIGS. 2A and 2B illustrate an exemplary implementation of an electronicdevice having a touch sensing display screen assembly and an opticalfingerprint sensor module positioned underneath the touch sensingdisplay screen assembly according to some embodiments.

FIGS. 3A and 3B illustrate an example of a device that implements theoptical fingerprint sensor module illustrated in FIGS. 2A and 2Baccording to some embodiments

FIGS. 4A and 4B show an exemplary implementation of an opticalfingerprint sensor module under (he display screen assembly forimplementing the design illustrated in FIGS. 2A and 2B according to someembodiments.

FIGS. 5A-5C illustrate signal generation tor the returned light from thesensing zone on the top sensing surface under two different opticalconditions to facilitate the understanding of the operation of anunder-screen optical fingerprint sensor module according to someembodiments.

FIGS. 6A-6C, 7, 8A-8B, 9, and 10A-10B illustrate example designs ofunder-screen optical fingerprint sensor modules according to someembodiments.

FIGS. 11A-11C illustrate imaging of the fingerprint sensing area on thetop transparent layer via an imaging module under different tilingconditions where an imaging device images the fingerprint sensing areaonto an optical sensor array and the imaging device may be opticallytransmissive or optically reflective according to some embodiments.

FIG. 12 is a flowchart illustrating an exemplary operation of afingerprint sensor for reducing or eliminating undesired contributionsfrom the background light in fingerprint sensing according to someembodiments.

FIG. 13 is a flowchart illustrating an exemplary process for operatingan under-screen optical fingerprint sensor module for capturing afingerprint pattern according to some embodiments.

FIGS. 14-16 illustrates exemplary operation processes for determiningwhether an object in contact with the LCD display screen is pan of afinger of a live person by illuminating the finger with light in twodifferent light colors according to some embodiments.

FIGS. 17A and 17B show an illustrative portable electronic device, and across-section of an illustrative display module for such a portableelectronic device, respectively, according to various embodiments.

FIGS. 18A-18C show views of an illustrative portion of a symmetricenhancement layer.

FIGS. 19A-19C show views of an illustrative portion of an asymmetricenhancement layer.

FIG. 20 shows an illustrative fingerprint sensing region overlaid on asingle enhancement film layer.

FIG. 21 shows an illustrative fingerprint sensing region overlaid on anenhancement panel having two enhancement film layers, stacked so thattheir respective micro-prism structures run in generally orthogonaldirections.

FIG. 22 shows an illustrative under-display optical sensing environment,according to various embodiments.

FIG. 23 shows an illustrative under-display optical sensing environmentwith off-axis sensing, according to various embodiments.

In the appended figures, similar components and/or features can have thesame reference label. Further, various components of the same type canbe distinguished by following the reference label by a second label thatdistinguishes among the similar components If only the first referencelabel is used in the specification, the description is applicable to anyone of the similar components having the same first reference labelirrespective of the second reference label.

DETAILED DESCRIPTION

In the following description, numerous specific details are provided fora thorough understanding of the present invention. However, it should beappreciated by those of skill in the art that the present invention maybe realized without one or more of these details. In other examples,features and techniques known in the art will not be described forpurposes of brevity.

Electronic devices or systems may be equipped with fingerprintauthentication mechanisms to improve the security for accessing thedevices. Such electronic devices or system may include, portable ormobile computing devices, e.g., smartphones, tablet computers,wrist-worn devices and other wearable or portable devices, largerelectronic devices or systems, e.g., personal computers in portableforms or desktop forms, ATMs, various terminals to various electronicsystems, databases, or information systems for commercial orgovernmental uses, motorized transportation systems includingautomobiles, boats, trains, aircraft and others.

Fingerprint sensing is useful in mobile applications and otherapplications that use or require secure access. For example, fingerprintsensing can be used to provide secure access to a mobile device andsecure financial transactions including online purchases. It isdesirable to include robust and reliable fingerprint sensing suitablefor mobile devices and other applications. In mobile, portable orwearable devices, it is desirable for fingerprint sensors to minimize oreliminate the footprint for fingerprint sensing given the limited spaceon those devices, especially considering the demands for a maximumdisplay area on a given device. Many implementations of capacitivefingerprint sensors must be implemented on lire top surface of a devicedue to the near-field interaction requirement of capacitive sensing.

Optical sensing modules can be designed to mitigate the above and otherlimitations in the capacitive fingerprint sensors and to achieveadditional technical advantages. For example, in implementing an opticalfingerprint sensing device, the light carrying fingerprint imagininginformation can be directed over distance lo an optical detector arrayof optical detectors for detecting the fingerprint without being limitedto the near-field sensing in a capacitive sensor. In particular, lightcarrying fingerprint imagining information can be directed to transmitthrough the top cover glass commonly used in many display screens suchas touch sensing screens and other structures and may be directedthrough folded or complex optical paths to reach the optical detectorarray, thus allowing for flexibility in placing an optical fingerprintsensor in a device that is not available for a capacitive fingerprintsensor. Optical fingerprint sensor modules based on the technologiesdisclosure herein can be an under-screen optical fingerprint sensormodule that is placed below a display screen to capture and detect lightfrom a finger placed on or above the top sensing surface of the screen.As disclosed herein, optical sensing can also be used to, in addition todetecting and sensing a fingerprint pattern, optically detect otherparameters associated with a user or a user action, such as whether adetected fingerprint is from a finger of a live person and to provideanti-spoofing mechanism, or certain biological parameters of the user.

I. OVERVIEW OF UNDER-DISPLAY OPTICAL SENSING MODULES

The optical sensing technology and examples of implementations describedin this disclosure provide an optical fingerprint sensor module thatuses, at least in part, the light from a display screen as theillumination probe light to illuminate a fingerprint sensing area on thetouch sensing surface of the display screen to perform one or moresensing operations based on optical sensing of such light. A suitabledisplay screen for implementing the disclosed optical sensor technologycan be based on various display technologies or configurations,including, a liquid crystal display (LCD) screen using a backlight toprovide white light illumination to the LCD pixels and matched opticalfilters to effectuate colored LCD pixels, or a display screen havinglight emitting display pixels without using backlight where eachindividual pixel generates light for forming a display image on thescreen such as an organic light emitting diode (OLED) display screens,or electroluminescent display screens. The specific examples providedbelow are directed to integration or under-screen optical sensingmodules with LCD screens and thus contain certain technical detailsassociated with LCD screens although various aspects of the disclosedtechnology are applicable to OLED screens and other display screens.

A portion of the light produced by a display screen for displayingimages necessarily passes through the top surface of the display screenin order to be viewed by a user. A finger in touch with or near the topsurface interacts with the light at the top surface to cause thereflected or scattered light at the surface area of the touch to carryspatial image information of the finger. Such reflected or scatteredlight carrying the spatial image information of the finger returns tothe display panel underneath the top surface. In touch sensing displaydevices, for example, the top surface is the touch sensing interfacewith the user and this interaction between the light for displayingimages and the user finger or hand constantly occurs but suchinformation-carrying light returning back to the display panel islargely wasted and is not used in various touch sensing devices. Invarious mobile or portable devices with touch sensing displays andfingerprint sensing functions, a fingerprint sensor tends to be aseparate device from the display screen, either placed on the samesurface of the display screen at a location outside the display screenarea such as in some models of Apple iPhones and Samsung smartphones, orplaced on the backside of a smartphone, such as some models of smartphones by Huawei, Lenovo, Xiaomi or Google, to avoid taking up valuablespace for placing a large display screen on the front side. Thosefingerprint sensors are separate devices from the display screens andthus need to be compact to save space for the display screens and otherfunctions while still providing reliable and fast fingerprint sensingwith a spatial image resolution above a certain acceptable level.However, the need to be compact and small for designing a fingerprintsensor and the need to provide a high spatial image resolution incapturing a fingerprint pattern are in direct conflict with each otherin many fingerprint sensors because a high spatial image resolution incapturing a fingerprint pattern in based on various suitable fingerprintsensing technologies (e.g., capacitive touch sensing or optical imaging)requires a large sensor area with a large number of sensing pixels.

The sensor technology and examples of implementations of the sensortechnology described in this disclosure provide an optical fingerprintsensor module that uses, at least in part, the light front a displayscreen as the illumination probe light to illuminate a fingerprintsensing area on the touch sensing surface of the display screen toperform one or more sensing operations based on optical sensing of suchlight in some implementations, or designated illumination or probe lightfor optical sensing from one or more designated illumination lightsources separate from the display light for optical sensing in otherimplementations, or background light for optical sensing in certainimplementations.

In the disclosed examples for integrating an optical sensing module to aLCD screen based on the disclosed optical sensor technology, the underLCD optical sensor can be used to detect a portion of the light that isused for displaying images in a LCD screen where such a portion of thelight for the display screen may be the scattered light, reflected lightor some stray light. For example, in some implementations, the imagelight of the LCD screen based on backlighting may be reflected orscattered back into the LCD display screen as returned light whenencountering an object such as a user linger or palm, or a user pointerdevice like a stylus. Such returned light can be captured for performingone or more optical sensing operations using the disclosed opticalsensor technology. Due to the use of the light from LCD screen foroptical sensing, an optical fingerprint sensor module based on thedisclosed optical sensor technology is specially designed to beintegrated to the LCD display screen in a way that maintains the displayoperations and functions of the LCD display screen without interferencewhile providing optical sensing operations and functions to enhanceoverall functionality, device integration and user experience of anelectronic device or system such as a smart phone, a tablet, or a mobileand/or wearable device.

In addition, in various implementations of the disclosed optical sensingtechnology, one or more designated probe light sources may be providedto produce additional illumination probe light for the optical sensingoperations by the under-LCD screen optical sensing module. In suchapplications, the light from the backlighting of the LCD screen and theprobe light from the one or more designated probe light sourcescollectively form the illumination light for optical sensing operations.

Regarding the additional optical sensing functions beyond fingerprintdetection, the optical sensing may be used to measure other parameters.For example, the disclosed optical sensor technology can measure apattern of a palm of a person given the large touch area available overthe entire LCD display screen (in contrast, some designated fingerprintsensors such as the fingerprint senor in the home button of Apple'siPhone/iPad devices have a rather small and designated off-screenfingerprint sensing area that is highly limited in the sensing area sizethat may not be suitable for sensing large patterns). For yet anotherexample, the disclosed optical sensor technology can be used not only touse optical sensing to capture and detect a pattern of a finger or palmthat is associated with a person, but also to use optical sensing orother sensing mechanisms to detect whether the captured or detectedpattern of a fingerprint or palm is from a live person's hand by a “livefinger” detection mechanism, which may be based on, for example, thedifferent optical absorption behaviors of the blood at different opticalwavelengths, the fact that a live person's finger tends to be moving orstretching due to the person's natural movement or motion (eitherintended or unintended) or pulsing when the blood flows through theperson's body in connection with the heartbeat. In one implementation,the optical fingerprint sensor module can detect a change in thereturned light from a finger or palm due to the heartbeat/blood flowchange and thus to detect whether there is a live heartbeat in theobject presented as a finger or palm. The user authentication can bebased on the combination of the both the optical sensing of thefingerprint/palm pattern and the positive determination of the presenceof a live person to enhance the access control. For yet another example,the optical fingerprint sensor module may include a sensing function formeasuring a glucose level or a degree of oxygen saturation based onoptical sensing in the returned light from a finger or palm. As yetanother example, as a person touches the LCD display screen, a change inthe touching force can be reflected in one or more ways, includingfingerprint pattern deforming, a change in the contacting area betweenthe finger and the screen surface, fingerprint ridge widening, or achange in the blood flow dynamics. Those and other changes can bemeasured by optical sensing based on the disclosed optical sensortechnology and can be used to calculate the touch force. This touchforce sensing can be used to add more functions to the opticalfingerprint sensor module beyond the fingerprint sensing.

With respect to useful operations or control features in connection withthe touch sensing aspect of the LCD display screen, the disclosedoptical sensor technology can provide triggering functions or additionalfunctions based on one or more sensing results from the opticalfingerprint sensor module to perform certain operations in connectionwith the touch sensing control over the LCD display screen. For example,the optical property of a finger skin (e.g., the index of refraction)tends to be different from other artificial objects. Based on this, theoptical fingerprint sensor module may be designed to selectively receiveand detect returned light that is caused by a finger in touch with thesurface of the LCD display screen while returned light caused by otherobjects would not be detected by the optical fingerprint sensor module.This object-selective optical detection can be used to provide usefuluser controls by touch sensing, such as waking up the smartphone ordevice only by a touch via a person's finger or palm while touches byother objects would not cause the device to wake up for energy efficientoperations and to prolong the battery use. This operation can beimplemented by a control based on the output of the optical fingerprintsensor module to control the waking up circuitry operation of the LCDdisplay screen which, the LCD pixels are put in a “sleep” mode by beingturned off (and the LCD backlighting is also turned off) while one ormore illumination light sources (e.g., LEDs) for the under-LCD paneloptical fingerprint sensor module arc turned on in a flash mode tointermittently emit flash light to the screen surface for sensing anytouch by a person's finger or palm. Under this design, the opticalfingerprint sensor module operates the one or more illumination lightsources to produce the “sleep” mode wake-up sensing light flashes sothat the optical fingerprint sensor module can detect returned light ofsuch wake-up sensing light caused by the finger touch on the LCD displayscreen and, upon a positive detection, the LCD backlighting and the LCDdisplay screen are turned on or “woken up”. In some implementations, thewake-up sensing light can be in the infrared invisible spectral range soa user will not experience any visual of a flash light. The LCD displayscreen operation can be controlled to provide an improved fingerprintsensing by eliminating background light for optical sensing of thefingerprint. In one implementation, for example, each display scan framegenerates a frame of fingerprint signals. If, two frames of fingerprintsignals with the display are generated in one frame when the LCD displayscreen is turned on and in the other frame when the LCD display screenis turned off, the subtraction between those two frames of signals canbe used to reduce the ambient background light influence. By operatingthe fingerprint sensing frame rate is at one half of the display framerate in some implementations, the background light noise in fingerprintsensing can be reduced.

An optical fingerprint sensor module based on the disclosed opticalsensor technology can be coupled to the backside of the LCD displayscreen without requiring creation of a designated area on the surfaceside of the LCD display screen that would occupy a valuable devicesurface real estate in some electronic devices such as a smartphone, atablet or a wearable device. This aspect of the disclosed technology canbe used to provide certain advantages or benefits in both device designsand product integration or manufacturing.

In some implementations, an optical fingerprint sensor module based onthe disclosed optical sensor technology can be configured as anon-invasive module that can be easily integrated to a display screenwithout requiring changing the design of the LCD display screen forproviding a desired optical sensing function such as fingerprint sensingIn this regard, an optical fingerprint sensor module based on thedisclosed optical sensor technology can be independent from the designof a particular LCD display screen design due to the nature of theoptical fingerprint sensor module the optical sensing of such an opticalfingerprint sensor module is by detecting the light that is emitted bythe one or more illumination light sources of the optical fingerprintsensor module and is returned from the top surface of the display area,and the disclosed optical fingerprint sensor module is coupled to thebackside of the LCD display screen as a under-screen optical fingerprintsensor module for receiving the returned light from the top surface ofthe display area and thus does not require a special sensing port orsensing area that is separate from the display screen area. Accordingly,such an under-screen optical fingerprint sensor module can be used tocombine with a LCD display screen to provide optical fingerprint sensingand other sensor functions on an LCD display screen without using aspecially designed LCD display screen with hardware especially designedfor providing such optical sensing. This aspect of the disclosed opticalsensor technology enables a wide range of LCD display screens insmartphones, tablets or other electronic devices with enhanced functionsfrom the optical sensing of the disclosed optical sensor technology.

For example, for an existing phone assembly design that does not providea separate fingerprint sensor as in certain Apple iPhones or SamsungGalaxy smartphones, such an existing phone assembly design can integratethe under-screen optical fingerprint sensor module as disclosed hereinwithout changing the touch sensing-display screen assembly to provide anadded on-screen fingerprint sensing function Because the disclosedoptical sensing does not require a separate designated sensing area orport as in the case of certain Apple iPhones/Samsung Galaxy phones witha front fingerprint senor outside the display screen area, or somesmartphones with a designated rear fingerprint sensor on the backsidelike in some models by Huawei, Xiaomi, Google or Lenovo, the integrationof the on-screen fingerprint sensing disclosed herein does not require asubstantial change to the existing phone assembly design or the touchsensing display module that has both the touch sensing layers and thedisplay layers. Based on the disclosed optical sensing technology inthis document, no external sensing port and no external hardware buttonare needed on the exterior of a device are needed for adding thedisclosed optical fingerprint sensor module for fingerprint sensing. Theadded optical fingerprint sensor module and the related circuitry areunder the display screen inside the phone housing and the fingerprintsensing can be conveniently performed on the same touch sensing surfacefor the touch screen.

For another example, due to the above described nature of the opticalfingerprint sensor module for fingerprint sensing, a smartphone thatintegrates such an optical fingerprint sensor module can be updated withimproved designs, functions and integration mechanism without affectingor burdening the design or manufacturing of the LCD display screens toprovide desired flexibility to device manufacturing andimprovements/upgrades in product cycles while maintaining theavailability of newer versions of optical sensing functions tosmartphones. tablets or other electronic devices using LCD displayscreens. Specifically, the touch sensing layers or the LCD displaylayers may be updated in the next product release without adding anysignificant hardware change for the fingerprint sensing feature usingthe disclosed under-screen optical fingerprint sensor module. Also,improved on-screen optical sensing for fingerprint sensing or otheroptical sensing functions by such an optical fingerprint sensor modulecan be added to a new product release by using a new version of theunder-screen optical fingerprint sensor module without requiringsignificant changes to the phone assembly designs, including addingadditional optical sensing functions.

The above and other features of the disclosed optical sensor technologycan be implemented to provide a new generation of electronic deviceswith improved fingerprint sensing and other sensing functions,especially for smartphones, tablets and other electronic devices withLCD display screens to provide various touch sensing operations andfunctions and to enhance the user experience in such devices Thefeatures for optical fingerprint sensor modules disclosed herein may beapplicable to various display panels based on different technologiesincluding both LCD and OLED displays The specific examples below aredirected to LCD display panels and optical fingerprint sensor modulesplaced under LCD display panels.

In implementations of the disclosed technical features, additionalsensing functions or sensing modules, such as a biomedical sensor, e.g.,a heartbeat sensor in wearable devices like wrist band devices orwatches, may be provided, in general, different sensors can be providedin electronic devices or systems to achieve different sensing operationsand functions.

The disclosed technology can be implemented to provide devices, systems,and techniques that perform optical sensing of human fingerprints andauthentication for authenticating an access attempt to a lockedcomputer-controlled device such as a mobile device or acomputer-controlled system, that is equipped with a fingerprintdetection module. The disclosed technology can be used for securingaccess to various electronic devices and systems, including portable ormobile computing devices such as laptops, tablets, smartphones, andgaming devices, and other electronic devices or systems such aselectronic databases, automobiles, bank ATMs, etc.

II. DESIGN EXAMPLES OF UNDER-DISPLAY OPTICAL SENSING MODULES

As described herein, embodiments provide large-sensing-areaimplementations of under-display optical sensing modules, such as forunder-screen optical fingerprint modules. For the sake of added clarityand context, examples are described of various designs for anunder-screen optical fingerprint sensor module for collecting an opticalsignal to the optical detectors and providing desired optical imagingsuch as a sufficient imaging resolution. These and other embodiments ofunder-display optical fingerprint sensing implementations are furtherdescribed in the following patent documents, which are herebyincorporated by reference in their entirety. U.S. patent applicationSer. No. 15/616,856; U.S. patent application Ser. No 15/421,249, U.S.patent application Ser. No. 16/190,138; U.S. patent application Ser. No.16/190,141; U.S. patent application Ser. No. 16/246,549; and U.S. patentapplication Ser. No. 16/427,269.

FIG. 1 is a block diagram of an example of a system 180 with afingerprint sensing module 180 including a fingerprint sensor 181 whichcan be implemented to include an optical fingerprint sensor based on theoptical sensing of fingerprints as disclosed in this document. Thesystem 180 includes a fingerprint sensor control circuit 184, and adigital processor 186 which may include one or more processors forprocessing fingerprint patterns and determining whether an inputfingerprint pattern is one for an authorized user. The fingerprintsensing system 180 uses the fingerprint sensor 181 to obtain afingerprint and compares the obtained fingerprint to a storedfingerprint to enable or disable functionality in a device or system 188that is secured by the fingerprint sensing system 180. In operation, theaccess to the device 188 is controlled by the fingerprint processingprocessor 186 based on whether the captured user fingerprint is from anauthorized user. As illustrated, the fingerprint sensor 181 may includemultiple fingerprint sensing pixels such as pixels 182A 182E thatcollectively represent at least a portion of a fingerprint. For example,the fingerprint sensing system 180 may be implemented at an ATM as thesystem 188 to determine the fingerprint of a customer requesting toaccess funds or other transactions. Based on a comparison of thecustomer's fingerprint obtained from the fingerprint sensor 181 to oneor more stored fingerprints, tie fingerprint sensing system 180 may,upon a positive identification, cause the ATM system 188 to gram therequested access to the user account, or, upon a negativeidentification, may deny the access. For another example, the device orsystem 188 may be a smartphone or s portable device and the fingerprintsensing system 180 is a module integrated to the device 188. For anotherexample, the device or system 188 may be a gate or secured entrance to afacility or home that uses the fingerprint sensor 181 to grant or denyentrance. For yet another example, the device or system 188 may be anautomobile or other vehicle that uses the fingerprint sensor 181 to linkto the start of the engine and to identify whether a person isauthorized to operate the automobile or vehicle.

As a specific example, FIGS. 2A and 2B illustrate one exemplaryimplementation of an electronic device 200 having a touch sensingdisplay screen assembly and an optical fingerprint sensor modulepositioned underneath the touch sensing display screen assembly. In thisparticular example, the display technology can be implemented by a LCDdisplay screen with backlight for optically illuminating the LCD pixelsor another display screen having light emitting display pixels withoutusing backlight (e.g., an OLED display screen). The electronic device200 can be a portable device such as a smartphone or a tablet and can bethe device 188 as shown in FIG. 1.

FIG. 2A shows the front side of the device 200 which may resemble somefeatures in some existing smartphones or tablets. The device screen ison the front side of the device 200 occupying either entirely, amajority or a significant portion of the front side space and thefingerprint sensing function is provided on the device screen, e.g., oneor more sensing areas for receiving a finger on the device screen. As anexample, FIG. 2A shows a fingerprint sensing zone in the device screenfor a finger to touch which may be illuminated as a visibly identifiablezone or area for a user to place a finger for fingerprint sensing. Sucha fingerprint sensing zone can function like the rest of the devicescreen for displaying images. As illustrated, the device housing of thedevice 200 may have, in various implementations, side facets thatsupport side control buttons that are common in various smartphones onthe market today. Also, one or more optional sensors may be provided onthe front side of the device 200 outside the device screen asillustrated by one example on the left upper comer of the device housingin FIG. 2A.

FIG. 2B shows an example of the structural construction of the modulesin the device 200 relevant to the optical fingerprint sensing disclosedin this document The device screen assembly shown in FIG. 2B includes,e.g., the touch sensing screen module with touch sensing layers on thetop, and a display screen module with display layers located underneaththe touch sensing screen module. An optical fingerprint sensor module iscoupled to, and located underneath, the display screen assembly moduleto receive and capture the returned light from the top surface of thetouch sensing screen module and to guide and image the returned lightonto an optical sensor array of optical sensing pixels or photodetectorswhich convert the optical image in the returned light into pixel signalsfor further processing. Underneath the optical fingerprint sensor moduleis the device electronics structure containing certain electroniccircuits for the optical fingerprint sensor module and other parts inthe device 200. The device electronics may be arranged inside the devicehousing and may include a part that is under the optical fingerprintsensor module as shown in FIG. 2B.

In implementations, the top surface of the device screen assembly can bea surface of an optically transparent layer serving as a user touchsensing surface to provide multiple functions, such as (1) a displayoutput surface through which the light carrying the display imagespasses through to reach a viewer's eyes. (2) a touch sensing interfaceto receive a user's touches for the touch sensing operations by thetouch sensing screen module, and (3) an optical interface for on-screenfingerprint sensing (and possibly one or more other optical sensingfunctions). This optically transparent layer can be a rigid layer suchas a glass or crystal layer or a flexible layer.

One example of a display screen is an LCD display having LCD layers anda thin film transistor (TFT) structure or substrate. A LCD display panelis a multi-layer liquid crystal display (LCD) module that includes LCDdisplay backlighting light sources (e.g., LED lights) emitting LCDillumination light for LCD pixels, a light waveguide layer to guide thebacklighting light, and LCD structure layers which can include, e.g., alayer of liquid crystal (LC) cells, LCD electrodes, transparentconductive ITO layer, an optical polarizer layer, a color filter layer,and a touch sensing layer. The LCD module also includes a backlightingdiffuser underneath the LCD structure layers and above the lightwaveguide layer to spatially spread the backlighting light forilluminating the LCD display pixels, and an optical reflector film layerunderneath the light waveguide layer to recycle backlighting lighttowards the LCD structure layers for improved light use efficiency andthe display brightness. For optical sensing, one or more separateillumination light sources are provided and are operated independentlyfrom the backlighting light sources of the LCD display module.

Referring to FIG. 2B, the optical fingerprint sensor module in thisexample is placed under the LCD display panel to capture the retunedlight from the top touch sensing surface and to acquire high resolutionimages of fingerprint patterns when user's ringer is in touch with asensing area on the top surface. In other implementations, the disclosedunder-screen optical fingerprint sensor module for fingerprint sensingmay be implemented on a device without the touch sensing feature.

FIGS. 3A and 3B illustrate an example of a device that implements theoptical fingerprint sensor module in FIGS. 2A and 2B. FIG. 3A shows across sectional view of a portion of the device containing theunder-screen optical fingerprint sensor module. FIG. 3B shows, on theleft, a view of the front side of the device with the touch sensingdisplay indicating a fingerprint sensing area on the lower part of thedisplay screen, and on the right, a perspective view of a part of thedevice containing the optical fingerprint sensor module that is underthe device display screen assembly. FIG. 3B also shows an example of thelayout of the flexible tape with circuit elements.

In the design examples in FIGS. 2A-2B, and 3A-3B, the opticalfingerprint sensor design is different from some other fingerprintsensor designs using a separate fingerprint sensor structure from thedisplay screen with a physical demarcation between the display screenand the fingerprint sensor (e.g., a button like structure in an openingof the top glass cover in some mobile phone designs) on the surface ofthe mobile device. In the illustrated designs here, the opticalfingerprint sensor for detecting fingerprint sensing and other opticalsignals are located under the top cover glass or layer (e.g., FIG. 3A)so that the top surface of the cover glass serves as the top surface ofthe mobile device as a contiguous and uniform glass surface across boththe display screen layers and the optical detector sensor that arevertically stacked and vertically overlap. This design example forintegrating optical fingerprint sensing and the touch sensitive displayscreen under a common and uniform surface provides benefits, includingimproved device integration, enhanced device packaging, enhanced deviceresistance to exterior elements, failure and wear and tear, and enhanceduser experience over the ownership period of the device.

Referring back to FIGS 2A and 2B, the illustrated under-screen opticalfingerprint sensor module for on-screen fingerprint sensing may beimplemented in various configurations. In one implementation, a devicebased on the above design can be structured to include a device screen athat provides touch sensing operations and includes a LCD display panelstructure for forming a display image, a top transparent layer formedover the device screen as an interface for being touched by a user forthe touch sensing operations and for transmitting the light from thedisplay structure to display images to a user, and an opticalfingerprint sensor module located below the display panel structure toreceive light that returns from the top transparent layer to detect afingerprint.

This device and other devices disclosed herein can be further configuredto include various features For example, a device electronic controlmodule can be included in the device to grant a user's access to thedevice if a detected fingerprint matches a fingerprint an authorizeduser. In addition, the optical fingerprint sensor module is configuredto, in addition to detecting fingerprints, also detect a biometricparameter different form a fingerprint by optical sensing to indicatewhether a touch at the top transparent layer associated with a detectedfingerprint is from a live person, and the device electronic centralmodule is configured to grant a user's access to the device if both (1)a detected fingerprint matches a fingerprint an authorized user and (2)the detected biometric parameter indicates the detected fingerprint isfrom a live person The biometric parameter can include, e.g., whetherthe finger contains a blood flow, or a heartbeat of a person.

For example, the device can include a device electronic control modulecoupled to the display panel structure to supply power to the lightemitting display pixels and to control image display by the displaypanel structure, and, in a fingerprint sensing operation, the deviceelectronic control module operates to turn off the light emittingdisplay pixels in one frame to and tum on the light emitting displaypixels in a next frame to allow the optical sensor array to capture twofingerprint images with and without the illumination by the lightemitting display pixels to reduce background light in fingerprintsensing.

For another example, a device electronic control module may be coupledto the display panel structure to supply power to the LCD display paneland to turn off power to the backlighting of the LCD display panel in asleep mode, and the device electronic control module may be configuredto wake up the display panel structure from the sleep mode when theoptical fingerprint sensor module detects the presence of a person'sskin at the designated fingerprint sensing region of the top transparentlayer. More specifically, in some implementations, the device electroniccontrol module can be configured to operate one or more illuminationlight sources in the optical fingerprint sensor module to intermittentlyemit light, while turning off power to the LCD display panel (in thesleep mode), to direct the intermittently emitted illumination light tothe designated fingerprint sensing region of the top transparent layerfor monitoring whether there is a person's skin in contact with thedesignated fingerprint sensing region for waking up the device from thesleep mode.

For another example, the device can include a device electronic controlmodule coupled to the optical fingerprint sensor module to receiveinformation on multiple detected fingerprints obtained from sensing atouch of a finger and the device electronic control module is operatedto measure a change in the multiple detected fingerprints and determinesa touch force that causes the measured change. For instance, the changemay include a change in the fingerprint image due to the touch force, achange in the touch area due to the touch force, or a change in spacingof fingerprint ridges.

For another example, the top transparent layer can include a designatedfingerprint sensing region for a user to touch with a finger forfingerprint sensing and the optical fingerprint sensor module below thedisplay panel structure can include a transparent block in contact withthe display panel substrate to receive light that is emitted from thedisplay panel structure and returned from the top transparent layer, anoptical sensor array that receives the light and an optical imagingmodule that images the received light in the transparent block onto theoptical sensor array. The optical fingerprint sensor module can bepositioned relative to the designated fingerprint sensing region andstructured to selectively receive returned light via total internalreflection at the top surface of the top transparent layer when incontact with a person's skin while not receiving the returned light fromthe designated fingerprint sensing region in absence of a contact by aperson's skin.

For yet another example, the optical fingerprint sensor module can bestructured to include an optical wedge located below the display panelstructure to modify a total reflection condition on a bottom surface ofthe display panel structure that interfaces with the optical wedge topermit extraction of light out of the display panel structure throughthe bottom surface, an optical sensor array that receives the light fromthe optical wedge extracted from the display panel structure, and anoptical imaging module located between the optical wedge and the opticalsensor array to image the light from the optical wedge onto the opticalsensor array.

FIGS. 4A and 4B show an example of one implementation of an opticalfingerprint sensor module under the display screen assembly forimplementing the design in FIGS. 2A and 2B. The device illustrated inFIGS. 4A and 4B includes a display assembly 423 with a top transparentlayer 431 formed over the device screen assembly 423 as an interface forbeing touched by a user for the touch sensing operations and fortransmitting the light from the display structure to display images to auser This top transparent layer 431 can be a cover glass or a crystalmaterial in some implementations. The device screen assembly 423 caninclude a LCD display module 433 under the top transparent layer 431.The LCD display layers allow partial optical transmission so light fromthe top surface can partially transmit through the LCD display layers toreach the under-LCD optical fingerprint sensor module. For example, LCDdisplay layers include electrodes and wiring structure optically actingas an array of holes and tight scattering objects. A device circuitmodule 435 may be provided under the LCD display panel to controloperations of the device and perform functions for the user to operatethe device.

The optical fingerprint sensor module 702 in this particularimplementation example is placed under LCD display module 433. One ormore illumination light sources, e.g., an illumination light source 436under the LCD display module 433 or/and another one or more illuminationlight sources located under the top cover glass 431, are provided forproviding the illumination light or probe light for the optical sensingby the optical fingerprint sensor module 702 and can be controlled toemit light to at least partially pass through the LCD display module 433to illuminate the fingerprint sensing zone 615 on the top transparentlayer 431 within the device screen area for a user to place a fingertherein for fingerprint identification. The illumination light from theone or more illumination light sources 436 can be directed to thefingerprint sensing area 615 on the top surface as if such illuminationlight is from a fingerprint illumination light zone 613. Another one ormore illumination light sources may be located under the top cover glass431 and may be placed adjacent to the fingerprint sensing area 615 onthe top surface to direct produced illumination light to reach the topcover glass 433 without passing through the LCD display module 433. Insome designs, one or more illumination light sources may be locatedabove the bottom surface of the lop cover glass 431 to direct producedillumination light to reach the fingerprint sensing region above the topsurface of the top cover glass 433 without necessarily passing throughthe top cover glass 431, e.g., directing illuminating the finger abovethe top cover glass 431.

As illustrated in FIG. 4A, a finger 445 is placed in the illuminatedfingerprint sensing zone 615 as the effective sensing zone forfingerprint sensing. A portion of the reflected or scattered light inthe zone 615 is directed into the optical fingerprint sensor moduleunderneath the LCD display module 433 and a photodetector sensing arrayinside the optical fingerprint sensor module receives such light andcaptures the fingerprint pattern information carried by the receivedlight. The one or more illumination light sources 436 are separate fromthe backlighting sources for the LCD display module and are operatedindependently from the backlighting light sources of the LCD displaymodule.

In this design of using one or more illumination light sources 436 toprovide the illumination light for optical fingerprint sensing, eachillumination light source 436 maybe controlled in some implementationsto turn on intermittently with a relatively low cycle to reduce thepower used for the optical sensing operations The fingerprint sensingoperation can be implemented in a two-step process in someimplementations first, the one or more illumination light sources 436are turned on in a flashing mode without turning on the LCD displaypanel to use the flashing light to sense whether a finger touches thesensing zone 615 and, once a touch in the zone 615 is detected, theoptical sensing module is operated to perform the fingerprint sensingbased on optical sensing and the LCD display panel may be turned on.

In the example in FIG. 4B, the under-screen optical fingerprint sensormodule includes a transparent block 701 that is coupled to the displaypanel to receive the returned light from the top surface of the deviceassembly, and an optical imaging block 702 that performs the opticalimaging and imaging capturing. Light from the one or more illuminationlight sources 436, after reaching the cover top surface, e.g., the covertop surface at the sensing area 615 where a user finger touches or islocated without touching the cover top surface, is reflected orscattered back from the cover top surface in a design in which theillumination light source 436 is located to direct the illuminationlight to first transmit through the top cover glass 431 to reach thefinger When fingerprint ridges in contact of the cover top surface inthe sensing area 615, the light reflection under the fingerprint ridgesis different, due to the presence of the skin or tissue of the finger incontact at that location, from the light reflection at another locationunder the fingerprint valley, where the skin or tissue of the finger isabsent. This difference in light reflection conditions at the locationsof the ridges and valleys in the touched finger area on the cover lopsurface forms an image representing an image or spatial distribution ofthe ridges and valleys of the touched section of the finger. Thereflection light is directed back towards the LCD display module 433,and, after passing through the small holes of the LCD display module433, reaches the interface with the low index optically transparentblock 701 of the optical fingerprint sensor module. The low indexoptically transparent block 701 is constructed to have a refractiveindex less than a refractive index of the LCD display panel so that thereturned light can be extracted out of the LCD display panel into theoptically transparent block 701. Once the returned light is receivedinside the optically transparent block 701, such received light entersthe optical imaging unit as pail of the imaging sensing block 702 and isimaged onto the photodetector sensing array or optical sensing arrayinside the block 702. The light reflection differences betweenfingerprint ridges and valleys create the contrast of the fingerprintimage. As shown in FIG. 4B, a control circuit 704 (e.g. amicrocontroller or MCU) is coupled to the imaging sensing block 702 andto other circuitry such as the device main processor 705 on a maincircuit board.

In this particular example, the optical light path design is structuredso that the illumination light enters the cover top surface within thetotal reflection angles on the top surface between the substrate and airinterface and, therefore, the reflected light is collected mosteffectively by the imaging optics and imaging sensor array in the block702. In this design, the image of the fingerprint ridge/valley areaexhibits a maximum contrast due to the total internal reflectioncondition at each finger valley location where the finger tissue doesnot touch the top cover surface of the top cover glass 431. Someimplementations of such an imaging system may have undesired opticaldistortions that would adversely affect the fingerprint sensing.Accordingly, the acquired image may be further corrected by a distortioncorrection during the imaging reconstruction in processing the outputsignals of the optical sensor array in the block 702 based on theoptical distortion profile along the light paths of the returned lightat the optical sensor array The distortion correction coefficients canbe generated by images captured at each photodetector pixel by scanninga test image pattern one line pixel at a time, through the whole sensingarea in both X direction lines and Y direction lines. This correctionprocess can also use images from tuning each individual pixel on one ata time, and scanning through the whole image area of the photodetectorarray. This correction coefficients only need to be generated one timeafter assembly of the sensor.

The background light from environment (e.g., sunlight or roomillumination light) may enter the image sensor through the LCD panel topsurface, and through holes in the LCD display assembly 433. Suchbackground light can create a background baseline in the interestedimages from a finger and thus may undesirably degrade the contrast of acaptured image. Different methods can be used to reduce this undesiredbaseline intensity caused by the background light. One example is totune on and off the illumination light source 436 at a certainillumination modulation frequency f and the image sensor accordinglyacquires the received images at the same illumination modulationfrequency by phase synchronizing the light source driving pulse andimage sensor frame. Under this operation, only one of the image phasescontain light from the tight source In implementing this technique, theimaging capturing can be timed to capture images with the illuminationlight on at even (or odd) frames while turning oil the illuminationlight at odd (or even) frames and, accordingly, subtracting even and oddframes can be used to obtain an image which is mostly formed by lightemitted from the modulated illumination light source with significantlyreduced background light. Based on this design, each display scan framegenerates a frame of fingerprint signals and two sequential frames ofsignals are obtained by turning on the illumination light in one frameand off in the other frame. The subtraction of adjacent frames can beused to minimize or substantially reduce the ambient background lightinfluence. In implementations, the fingerprint sensing frame rate can beone half of the display frame rate.

In the example shown in FIG. 4B, a portion of the light from the one ormore illumination light sources 436 may also go through the cover topsurface and enter the finger tissues. This part of the illuminationlight is scattered around and a part of this scattered light may beeventually collected by the imaging sensor array in the opticalfingerprint sensor module 702 The light intensity of this scatteredlight b a result of interacting with the inner tissues of the finger andthus depends on the finger's skin color, the blood concentration in thefinger tissue or the inner finger tissues. Such information of thefinger is carried by this scattered light on the finger, is useful forfingerprint sensing, and can be detected as part of the fingerprintsensing operation. For example, the intensity of a region of usersfinger image can be integrated in detection for measuring or observingin increase or decrease in the blood concentration that is associatedwith or depends on the phase of the user's heart-beat. This signaturecan be used to determine the user's heart beat rate, to determine if theuser's finger is a live finger, or to provide a spoof device with afabricated fingerprint pattern. Additional examples of using informationin light carrying information on the inner tissues of a finger areprovided in later sections of this patent document.

The one or more illumination light sources 436 in FIG. 4B can bedesigned to emit illumination light of different colors or wavelengthsin some designs and the optical fingerprint sensor module can capturereturned light from a person's finger at the different colors orwavelengths. By recording the corresponding measured intensity of thereturned light at the different colors or wavelengths, informationassociated with the user's skin color, the blood flow or inner tissuestructures inside the finger can be measured Of determined. As anexample, when a user registers a finger for fingerprint authenticationoperation, the optical fingerprint sensor can be operated to measure theintensity of the scatter light from the finger at two different colorsor illumination light wavelengths associated with light color A andlight color B, as intensities Ia and Ib, respectively. The ratio ofIa/Ib could be recorded to compare with later measurement when theuser's finger is placed on the sensing area on the top sensing surfaceto measure the fingerprint. This method can be used as part of thedevice's anti spoofing system to reject a spoof device that isfabricated with a fingerprint emulating or being identical to a user'sfingerprint but may not match user's skin color or other biologicalinformation of the user.

The one or more illumination light sources 436 can be controlled by thesame electronics 704 (e.g., MCU) for controlling the image sensor arrayin the block 702. The one or more illumination light sources 436 can bepulsed for a short time (e.g., at a low duty cycle) to emit lightintermittently and to provide pulse light for image sensing. The imagesensor array can be operated to monitor the light pattern at the samepulse duty cycle. If there is a human finger touching the sensing area615 on the screen, the image that is captured at the imaging sensingarray in the block 702 can be used to detect the touching event. Thecontrol electronics or MCU 704 connected to the image sensor army in theblock 702 can be operated to determine if the touch is by a human fingertouch. If it is confirmed that it is a human finger touch event, the MCU704 can be operated to wake up the smartphone system, turn on the one ormore illumination light sources 436 for performing the opticalfingerprint sensing), and use the normal mode to acquire a fullfingerprint image. The image sensor array in the block 702 sends theacquired fingerprint image to the smartphone main processor 705 whichcan be operated to match the captured fingerprint image to theregistered fingerprint database. If there is a match, the smartphoneunlocks the phone to allow a user to access the phone and start thenormal operation, if the captured image is not matched, the smartphoneproduces a feedback to user that the authentication is failed andmaintains the locking status of the phone The user may try to go throughthe fingerprint sensing again, or may input a passcode as an alternativeway to unlock the phone.

In the example illustrated in FIGS. 4A and 4B, the under-screen opticalfingerprint sensor module uses the optically transparent block 701 andthe imaging sensing block 702 with the photodetector sensing array tooptically image the fingerprint pattern of a touching finger in contactwith the top surface of the display screen onto the photodetectorsensing array. The optical imaging axis or detection axis 625 from thesensing zone 615 to the photodetector array in the block 702 isillustrated in FIG. 4B for the illustrated example. The opticallytransparent block 701 and the front end of the imaging sensing block 702before the photodetector sensing array forma a bulk imaging module toachieve proper imaging for the optical fingerprint sensing Due to theoptical distortions in this imaging process, a distortion correction canbe used to achieve the desired imaging operation.

In the optical sensing by the under-screen optical fingerprint sensormodule in FIGS. 4A and 4B and other designs disclosed herein, theoptical signal from the sensing zone 615 on the top transparent layer431 to the under-screen optical fingerprint sensor module includedifferent light components.

FIGS. 5A-5C illustrate signal generation for the returned light from thesensing zone 615 under different optical conditions to facilitate theunderstanding of the operation of the under-screen optical fingerprintsensor module The light that enters into the finger, either from theillumination light source or from other light sources (e.g., backgroundlight) can generate internally scattered light in tissues below thefinger surface, such as the scattered light 191 in FIGS, 5A-5C. Suchinternally scattered light in tissues below the finger surface canpropagate through the internal tissues of the finger and subsequentlytransmits through the finger skin to enter the (op transparent layer 431carrying certain information is not carried by light that is scattered,refracted or reflected by the finger surface, e.g., information onfinger skin color, the blood concentration or flow characteristicsinside the finger, or an optical transmissive pattern of the finger thatcontains both (1) a two-dimensional spatial pattern of external ridgesand valleys of a fingerprint (2) an internal fingerprint patternassociated with internal finger tissue structures that give rise to theexternal ridges and valleys of a linger.

FIG. 5A shows an example of how illumination light from the one or moreillumination light sources 436 propagates through the OLED displaymodule 433, after transmitting through the lop transparent layer 431,and generates different returned light signals including light signalsthat carry fingerprint pattern information to the under-screen opticalfingerprint sensor module. For simplicity, two illumination rays 80 and82 at two different locations are directed to the top transparent layer431 without experiencing total reflection at the interfaces of the toptransparent layer 431. Specifically, the illumination light rays 80 and82 are perpendicular or nearly perpendicular to the top layer 431. Afinger 60 is in contact with the sensing zone 615 on the e toptransparent layer 431. As illustrated, the illumination light beam SOreaches to a finger ridge in contact with the top transparent layer 431after transmitting through the top transparent layer 431 to generate thelight beam 183 in the finger tissue and another light beam 181 backtowards the LCD display module 433. The illumination light beam 82reaches to a finger valley located above the top transparent layer 431after transmitting through the top transparent layer 431 to generate thereflected light beam 185 from the interface with the top transparentlayer 431 back towards the LCD display module 433, a second light beam189 that enters the finger tissue and a third light beam 187 reflectedby the finger valley.

In the example in FIG. 5A, it is assumed that the finger skin'sequivalent index of refraction is about 1.44 at 550 nm and the coverglass index of refraction is about 1.51 for the top transparent layer431. The finger ridge-cover glass interface reflects part of the beam 80as reflected light 181 to bottom layers 524 below-he LCD display module433. The reflectance can be low, e.g., about 0.1% in some LCD panels.The majority of the light beam 80 becomes the beam 183 that transmitsinto the finger tissue 60 which causes scattering of the light 183 toproduce the returned scattered light 191 towards the LCD display module433 and the bottom layers 524. The flattering of the transmitted lightbeam 189 from the LCD pixel 73 in the finger tissue also contributes tothe returned scattered light 191.

The beam 82 at the finger skin valley location 63 is reflected by thecover glass surface. In some designs, for example, the reflection may beabout 3.5% as the reflected light 185 towards bottom layers 524, and thefinger valley surface may reflect about 3.3% of the incident light power(light 187) to bottom layers 524 so that the total reflection may beabout 6.8%. The majority light 189 is transmitted into the fingertissues 60. Part of the light power in the transmitted light 189 in thefigure tissue is scattered by the tissue to contribute to the scatteredlight 191 towards and into the bottom layers 524.

Therefore, in the example in FIG. 5A, the light reflections from variousinterface or surfaces at finger valleys and finger ridges of a touchingfinger are different and the reflection ratio difference carries thefingerprint map information and can be measured to extract thefingerprint pattern of the portion that is in contact with the toptransparent layer 431 and is illuminated the OLED light.

FIGS. 5B and 5C illustrate optical paths of two additional types ofillumination light rays at the top surface tinder different conditionsand at different positions relative to valleys or ridges of a finger,including under a total reflection condition at the interface with thetop transparent layer 431. The illustrated illumination light raysgenerate different returned light signals including light signals thatcarry fingerprint pattern information to the under-screen opticalfingerprint sensor module. It is assumed that the cover glass 431 andthe LCD display module 433 are glued together without any air gap inbetween so that illumination light with a large incident angle to thecover glass 431 will be totally reflected at the cover glass-airinterface. FIGS. 5A, SB and 5C illustrate examples of three differentgroups divergent light beams: (1) central beams 82 with small incidentangles to the cover glass 431 without the total reflection (FIG. 5A),(2) high contrast beams 201, 202, 211, 212 that are totally reflected atthe cover glass 431 when nothing touches the cover glass surface and canbe coupled into finger tissues when a finger touches the cover glass 431(FIGS. 5B and 5C), and (3) escaping beams having very large incidentangles that are totally reflected at the cover glass 431 even at alocation where the finger issue is in contact.

For the central light beams 82, the cover glass surface in some designsmay reflect about 0.1%-3.5% to light beam 185 that is transmitted intobottom layers 524, the finger skin may reflect about 0.1%-3.3% to lightbeam 187 that is also transmitted into bottom layers 524 The reflectiondifference is dependent on whether the light beams 82 meet with fingerskin ridge 61 or valley 63. The rest light beam 189 is coupled into thefinger tissues 60.

For high contrast light beams 201 and 202 meeting the local totallyinternal reflection condition, the cover glass surface reflects nearly100% to light beams 205 and 206 respectively if nothing touches thecover glass surface. When the finger skin ridges touch the cover glasssurface and at light beams 201 and 202 positions, most of the lightpower may be coupled into the finger tissues 60 by light beams 203 and204.

For high contrast light beams 211 and 212 meeting the local totallyinternal reflection condition, the cover glass surface reflects nearly100% to light beams 213 and 214 respectively if nothing touches thecover glass surface When the finger touches the cover glass surface andthe finger skin valleys happen to be at light beams 211 and 212positions, no light power is coupled into finger tissues 60.

As illustrated in FIG. 5A, a portion of the illumination light that iscoupled into finger tissues 60 tends to experience random scattering bythe inner finger tissues to form low-contrast light 191 and part of suchlow-contrast light 191 can pass through the LCD display module 433 toreach to the optical fingerprint sensor module. This portion of lightcaptured by optical fingerprint sensor module contains additionalinformation on the finger skin color, blood characteristics and thefinger inner tissue structures associated with the fingerprint.Additional features for using internally scattered light in tissuesbelow the finger surface in optical sensing will be explained in laterpart of this patent document, such as obtaining an optical transmissivepattern of the finger that contains both (1) a two-dimensional spatialpattern of external ridges and Valleys of a fingerprint (2) an internalfingerprint pattern associated with internal finger tissue structuresthat give rise to the external ridges and valleys of a finger.Therefore, in high contrast light beams illuminated area, finger skinridges and valleys cause different optical reflections and thereflection difference pattern carries the fingerprint patterninformation. The high contrast fingerprint signals can be achieved bycomparing the difference.

The disclosed under-screen optical sensing technology can be in variousconfigurations to optically capture fingerprints based on the designillustrated in FIGS. 2A and 2B. For example, the specific implementationin FIG. 4B based on optical imaging by using a bulk imaging module inthe optical sensing module can be implemented in various configurations.

FIGS. 6A-6C show an example of an under-screen optical fingerprintsensor module based on optical imaging via a lens for capturing afingerprint from a finger 445 pressing on the display cover glass 423.FIG. 6C is an enlarged view of the optical fingerprint sensor modulepart shown in FIG. 6B. The under-screen. optical fingerprint sensormodule as shown in FIG. 6B is placed under the LCD display module 433includes an optically transparent spacer 617 that is engaged to thebottom surface of the LCD display module 433 to receive the returnedlight from the sensing zone 615 on the top surface of the toptransparent layer 431, an imaging lens 621 that is located between andspacer 617 and the photodetector array 623 lo image the receivedreturned light from the sensing zone 615 onto tie photodetector array623. Different from FIG. 4B showing an example of an optical projectionimaging system without a lens, the example of the imaging design in FIG.6B used the imaging lens 621 to capture the fingerprint image at thephotodetector array 623 and enables an image reduction by the design ofthe imaging lens 621. Similar to the imaging system in the example inFIG. 4B to some extent, this imaging system in FIG. 6B for the opticalfingerprint sensor module can experience image distortions and asuitable optical correction calibration can be used to reduce suchdistortions, e.g., the distortion correction methods described for thesystem in FIG. 4B.

Similar to the assumptions in FIGS. 5A-5C, it is assumed that the fingerskin's equivalent index of refraction to be about 1.44 at 550 nm and abare cover glass index of refraction to be about 1.51 for the coverglass 423. When the OLED display module 433 is glued onto the coverglass 431 without any air gap, the total inner reflection happens inlarge angles at or larger than the critical incident angle for theinterface. The total reflection incident angle is about 41.8° if nothingis in contact with the cover glass top surface, and the total reflectionangle is about 73.7° if the finger skin touches the cover glass topsurface. The corresponding total reflection angle difference is about31.9°.

In this design, the micro lens 621 and the photodiode array 623 define aviewing angle θ for capturing the image of a contact finger in thesensing zone 615. This viewing angle can be aligned properly bycontrolling the physical parameters or configurations in order to detecta desired pan of the cover glass surface in the sensing zone 615. Forexample, the viewing angle may be aligned to detect the total innerreflection of the LCD display assembly. Specifically, the viewing angleθ is aligned to sense the effective sensing zone 615 on the cover glasssurface. The effective sensing cover glass surface 615 may be viewed asa mirror so that the photodetector array effectively detects an image ofthe fingerprint illumination light zone 613 in the LCD display that isprojected by the sensing cover glass surface 615 onto the photodetectorarray. The photodiode/photodetector array 623 can receive (he image ofthe zone 613 that is reflected by the sensing cover glass surface 615.When a finger touches the sensing zone 615, some of the light can becoupled into the fingerprint's ridges and this will cause thephotodetector array to receive light from the location of the ridges toappear as a darker image of the fingerprint. Because the geometries ofthe optical detection path are known, the fingerprint image distortioncaused in the optical path in the optical fingerprint sensor module canbe corrected

Consider, as a specific example, that the distance H in FIG. 6B from thedetection module central axis to the cover glass top surface is 2 mmThis design can directly cover 5 mm of an effective sensing zone 615with a width Wc on the cover glass. Adjusting the spacer 617 thicknesscan adjust the detector position parameter H, and the effective sensingzone width Wc can be optimized. Because H includes the thickness of thecover glass 431 and the display module 433, the application designshould take these layers into account. The spacer 617, the micro lens621, and the photodiode array 623 can be integrated under the colorcoating 619 on the bottom surface of the top transparent layer 431.

FIG. 7 shows an example of further design considerations of the opticalimaging design for the optical fingerprint sensor module shown in FIGS.6A-6C by using a special spacer 618 to replace the spacer 617 in FIGS.6B-6C to increase the size of the sensing area 615. The spacer 618 isdesigned with a width Ws and thickness is Ms to have a low refractionindex (RI) ns, and is placed under the LCD display module 433, e.g.,being attached (e.g., glued) to the bottom surface the LCD displaymodule 433 The end facet of the spacer 618 is an angled or slanted facetthat interfaces with the micro lens 621. This relative position of thespacer and the lens is different from FIGS. 6B-6C, where the lens isplaced underneath the spacer 617. The micro lens 621 and a photodiodearray 623 are assembled into the optical detection module with adetection angle width θ. The detection axis 625 is bent due to opticalrefraction at the interface between the spacer 618 and display module433 and at the interface between the cover glass 431 and the air. Thelocal incident angle ϕ1 and ϕ2 are decided by the refractive indicesRIs, ns, nc, and na of the materials for the components.

If nc is greater than ns, ϕ1 is greater than ϕ2. Thus, the refractionenlarges the sensing width Wc. For example, assuming the finger skin'sequivalent RI is about 1.44 at 550 nm and the cover glass index RI isabout 1.51, the total reflection incident angle is estimated to be about41.8° if nothing touches the cover glass top surface, and the totalreflection angle is about 73.7° if the finger skin touches the coverglass top surface. The corresponding total reflection angle differenceis about 31.9°. If the spacer 618 is made of same material of the coverglass, and the distance from the detection module center to the coverglass top surface is 2 mm, if detection angle width is θ=31.9°, theeffective sensing area width Wc is about 5 mm The corresponding centralaxis's focal incident angle is ϕ1=ϕ2=57.75°. If the material for thespecial spacer 618 has a refractive index ns about 1.4, and Hs is 1.2 mmand the detection module is tilted at ϕ1=70°. The effective sensing areawidth is increased to be greater than 6.5 mm. Under those parameters,the detection angle width in the cover glass is reduced to 19°.Therefore, the imaging system for the optical fingerprint sensor modulecan be designed to desirably enlarge the size of the sensing area 615 onthe top transparent layer 431.

the refractive index RI of the special spacer 618 is designed to besufficiently low (e.g., to use MgF₂, CaF₂, or even air to form thespacer), the width Wc of the effective sensing area 615 is no longerlimited by the thickness of the cover glass 431 and the display module433. This property provides desired design flexibility. In principle, ifthe detection module has a sufficient resolution, the effective sensingarea may even be increased to cover the entire display screen.

Since the disclosed optical sensor technology can be used to provide alarge sensing area for capturing a pattern, the disclosed under-screenoptical fingerprint sensor modules may be used to capture and detect notonly a pattern of a finger but a larger size patter such a person's palmthat is associated with a person for user authentication.

FIGS. 8A-8B show an example of further design considerations of theoptical imaging design for the optical fingerprint sensor module shownin FIG. 7 by setting the detection angle θ′ of the photodetector arrayrelative in the display screen surface and the distance L between thelens 621 and the spacer 618. FIG. 8A shows a cross-sectional view alongthe direction perpendicular to the display screen surface, and FIG. 8Bshows a view of the device from either the bottom or top of the displacescreen. A filling material 618 c can be used to fill the space betweenthe lens 621 and the photodetector array 623. For example, the fillingmaterial 618 c can be same material of the special spacer 618 or anotherdifferent material. In some designs, the filling material 618 c may theair space.

FIG. 9 shows another example of an under-screen optical fingerprintsensor module based on the design in FIG. 7 where one or moreillumination light sources 614 are provided to illuminate the topsurface sensing zone 615 for optical fingerprint sensing. Theillumination light sources 614 may be of an expanded type, or be acollimated type so that all the points within the effective sensing zone615 is illuminated. The illumination light sources 614 may be a singleelement light source or an array of fight sources.

FIGS. 10A-10B show an example of an under-screen optical fingerprintsensor module that uses an optical coupler 628 shaped as a thin wedge toimprove the optical detection at the optical sensor array 623. FIG. 10Ashows a cross section of the device structure with an under-screenoptical fingerprint sensor module for fingerprint sensing and FIG. 10Bshows a top view of the device screen. The optical wedge 628 (with arefractive index ns) is located below the display panel structure tomodify a total refection condition on a bottom surface of the displaypanel structure that interfaces with the optical wedge 628 to permitextraction of light out of the display panel structure through thebottom surface The optical sensor array 623 receives the light from theoptical wedge 628 extracted from the display panel structure and theoptical imaging module 621 is located between the optical wedge 628 andthe optical sensor array 623 to image the light from the optical wedge628 onto the optical sensor array 623. In the illustrated example, theoptical wedge 628 includes a slanted optical wedge surface facing theoptical imaging module and the optical sensing array 623. Also, asshown, there is a free space between the optical wedge 628 and theoptical imaging module 621.

If the light is totally reflected at the sensing surface of the coverglass 431, the reflectance is 100%, of the highest efficiency. However,the light will also be totally reflected at the LCD bottom surface 433 bif it is parallel to the cover glass surfaces. The wedge coupler 628 isused to modify the local surface angle so that the light can be coupledout for the detection at the optical sensor stray 623. The micro holesin the LCD display module 433 provide the desired light propagation pathfor light to transmit through the LCD display module 433 for theunder-screen optical sensing The actual light transmission efficiencymay gradually be reduced if the light transmission angle becomes toolarge or when the TFT layer becomes too thick. When the angle is closeto the total reflection angle, namely about 41.8° when the cover glassrefractive index is 1.5, the fingerprint image looks good. Accordingly,the wedge angle of the wedge coupler 628 may be adjusted to be of acouple of degrees so that the detection efficiency can be increased oroptimized. If the cover glass' refractive index is selected to behigher, the total reflection angle becomes smaller For example, if thecover glass is made of Sapphire which refractive index is about 1.76,the total reflection angle is about 34.62°. The detection lighttransmission efficiency in the display is also improved. Therefore, thisdesign of using a thin wedge to set the detection angle to be higherthan the total reflection angle, and/or to use high refractive indexcover glass material to improve the detection efficiency.

In some under-screen optical fingerprint sensor module designs (e.g.,those illustrated in FIGS. 6A-6C, 7, 8A, 8B, 9, 10A, and 10B), thesensing area 615 on the top transparent surface is not vertical orperpendicular to the detection axis 625 of the optical fingerprintsensor module so that the image plane of the sensing area is also notvertical or perpendicular to the detection axis 625. Accordingly, theplane of tie photodetector array 623 can be tilted relative thedetection axis 625 to achieve high quality imaging at the photodetectorarray 623.

FIGS. 11A-11C show three example configurations for this tilting. FIG.11A shows the sensing area 615 a is tilted and is not perpendicular thedetection axis 625. In FIG. 11B, the sensing area 615 b is aligned to beon the detection axis 625, such that its image plane will also belocated on the detection axis 625. In practice, the lens 621 can bepartially cut off so as to simplify the package. In variousimplementations, the micro lens 621 can also be of transmission type orreflection type. For example, a specified approach is illustrated inFIG. 11C. The sensing area 615 c is imaged by an imaging mirror 621 a. Aphotodiode array 623 b is aligned to detect the signals.

In the above designs where the lens 621 is used, the lens 621 can bedesigned to have an effective aperture that is larger than the apertureof the holes in the LCD display layers that allow transmission of lightthrough the LCD display module for optical fingerprint sensing. Thisdesign can reduce the undesired influence of the wiring structures andother scattering objects in the LCD display module.

FIG. 12 shows an example of an operation of the fingerprint sensor forreducing or eliminating undesired contributions from the backgroundlight in fingerprint sensing. The optical sensor array can be used tocapture various frames and the captured frames can be used to performdifferential and averaging operations among multiple frames to reducethe influence of the background light. For example, in frame A, theillumination light source for optical fingerprint sensing is turned onto illuminate the finger touching area, in frame B the illumination ischanged or is turned off. Subtraction of the signals of frame B from thesignals of frame A can be used in the image processing to reduce theundesired background light influence.

The undesired background light in the fingerprint sensing may also bereduced by providing proper optical filtering in the light path. One ormore optical filters may be used to reject the environment lightwavelengths, such as near IR and partial of the red light etc. In someimplementation, such optical filter coatings may be made on the surfacesof the optical parts, including the display bottom surface, prismsurfaces, sensor surface etc. For example, human fingers absorb most ofthe energy of the wavelengths under ˜580 nm, if one or more opticalfilters or optical filtering coatings can be designed to reject light inwavelengths from 580 nm to infrared, undesired contributions to theoptical detection in fingerprint sensing from the environment light maybe greatly reduced.

FIG. 13 shows an example of an operation process for correcting theimage distortion in the optical fingerprint sensor module At step 1301,the one or more illumination light sources are controlled and operatedto emit light in a specific region, and the light emission of suchpixels is modulated by a frequency F. Ate step 1302, an imaging sensorunder the display panel is operated to capture the image at frame rateat same frequency F. In the optical fingerprint sensing operation, afinger is placed on top of the display panel cover substrate and thepresence of the finger modulates the light reflection intensity of thedisplay panel cover substrate top surface. The imaging sensor under thedisplay captures the fingerprint modulated reflection light pattern. Atstep 1303, the demodulation of the signals from image sensors issynchronized with the frequency F, and the background subtraction isperformed. The resultant image has a reduced background light effect andincludes images from pixel emitting tights. At step 1304, the captureimage is processed and calibrated to correct image system distortions.At step 1305, the corrected image is used as a human fingerprint imagefor user authentication.

The same optical sensors used for capturing the fingerprint of a usercan be used also to capture the scattered light from the illuminatedfinger as shown by the back scattered light 191 in FIG. 5A. The detectorsignals from the back scattered light 191 in FIG. 5A in a region ofinterest can be integrated to produce an intensity signal. The intensityvariation of this intensity signal is evaluated to determine otherparameters beyond the fingerprint pattern, e.g., the heart rate of theuser or inner topological tissues of a finger associated with theexternal fingerprint pattern.

The above fingerprint sensor may be hacked by malicious individuals whocan obtain the authorized user's fingerprint, and copy the stolenfingerprint pattern on a carrier object that resembles a human lingerSuch unauthorized fingerprint patterns may be used on the fingerprintsensor to unlock the targeted device. Hence, a fingerprint pattern,although a unique biometric identifier, may not be by itself acompletely reliable or secure identification. The under-screen opticalfingerprint sensor module can also be used to as an opticalanti-spoofing sensor for sensing whether an input object withfingerprint patterns is a finger from a living person and fordetermining whether a fingerprint input is a fingerprint spoofingattack. This optical anti-spoofing sensing function can be providedwithout using a separate optical sensor. The optical anti-spoofing canprovide high-speed responses without compromising the overall responsespeed of the fingerprint sensing operation.

FIG. 14 shows exemplary optical extinction coefficients of materialsbeing monitored in blood where the optical absorptions are differentbetween the visible spectral range e.g., red light at 660 nm and theinfrared range, e.g., IR light at 940 nm. By using probe light toilluminate a finger at a first visible wavelength (Color A) and a seconddifferent wavelength such as an infrared (IR) wavelength (Color B), thedifferences in the optical absorption of the input object can becaptured determine whether the touched object is a finger from a liveperson. The one or more illumination light sources for providing theillumination for optical sensing can be used to emit light of differentcolors to emit probe or illumination light at least two differentoptical wavelengths to use the different optical absorption behaviors ofthe blood for live finger detection. When a person' heart beats, thepulse pressure pumps the blood to flow in the arteries, so theextinction ratio of the materials being monitored in the blood changeswith the pulse. The received signal carries the pulse signals. Theseproperties of the blood can be used to detect whether the monitoredmaterial is a live-fingerprint or a fake fingerprint.

FIG. 15 shows a comparison between optical signal behaviors in thereflected light from a nonliving material (e.g., a fake finger or aspoof device with a fabricated fingerprint pattern) and a live finger.The optical fingerprint sensor can also operate as a heartbeat sensor tomonitor a living organism. When two or more wavelengths of the probelight are detected, the extinction ratio difference can be used toquickly determine whether the monitored material is a living organism,such as live fingerprint. In the example shown in FIG. 15, probe lightat different wavelengths were used, one at a visible wavelength andanother at an IR wavelength as illustrated in FIG. 14.

When a nonliving material touches the top cover glass above thefingerprint sensor module, the received signal reveals strength levelsthat are correlated to the surface pattern of the nonliving material andthe received signal does not contain signal components associated with afinger of a living person. However, when a finger of a living persontouches the top cover glass, the received signal reveals signalcharacteristics associated with a living person, including obviouslydifferent strength levels because the extinction ratios are differentfor different wavelengths. This method does not take long time todetermine whether the touching material is a part of a living person. InFIG. 15, the pulse-shaped signal reflects multiple touches instead ofblood pulse. Similar multiple touches with a nonliving material does notshow the difference caused by a living finger.

This optical sensing of different optical absorption behaviors of theblood at different optical wavelengths can be performed in a shortperiod for live finger detection and can be faster than opticaldetection of a person's heart beat using the same optical sensor.

In LCD displays, the LCD backlighting illumination light is white lightand thus contains light at both the visible and IR spectral ranges forperforming the above live finger detection at the optical fingerprintsensor module. The LCD color filters in the LCD display module can beused to allow the optical fingerprint sensor module to obtainmeasurements in FIGS. 14 and 15. In addition, the designated lightsources 436 for producing the illumination light for optical sensing canbe operated to emit probe light at the selected visible wavelength andIR wavelength at different times and the reflected probe light at thetwo different wavelengths is captured by the optical detector array 623to determine whether touched object is a live finger based on the aboveoperations shown in FIGS. 14 and 15. Notably, although the reflectedprobe light at the selected visible wavelength and IR wavelength atdifferent times may reflect different optical absorption properties ofthe blood, the fingerprint image is always captured by both the probelight the selected visible wavelength and the probe light at the IRwavelength at different times. Therefore, the fingerprint sensing can bemade at both the visible wavelength and IR wavelength.

FIG. 16 shows an example of an operation process for determining whetheran object in contact with the LCD display screen is part of a finger ofa live person by operating the one or more illumination light sourcesfor optical sensing to illuminate the finger with light in two differentlight colors.

For yet another example, the disclosed optical sensor technology can beused to detect whether the captured or detected pattern of a fingerprintor palm is from a live person's hand by a “live finger” detectionmechanism by other mechanisms other than the above described differentoptical absorptions of blood at different optical wavelengths. Forexample, a live person's finger tends to be moving or stretching due tothe person's natural movement or motion (either intended or unintended)or pulsing when the blood flows through the person's body in connectionwith the heartbeat. In one implementation, the optical fingerprintsensor module can detect a change in the returned light from a finger orpalm due to the heartbeat/blood flow change and thus to detect whetherthere is a live heartbeat in the object presented as a finger or palm.The user authentication can be based on the combination of the both theoptical sensing of the fingerprint/palm pattern and the positivedetermination of the presence of a live person to enhance the accesscontrol. For yet another example, as a person touches the LCD displayscreen, a change in the touching force can be reflected in one or moreways, including fingerprint pattern deforming, a change in thecontacting area between the finger and the screen surface. fingerprintridge widening, or a change in the blood flow dynamics. Those and otherchanges can be measured by optical sensing based on the disclosedoptical sensor technology and can be used to calculate the touch force.This touch force sensing can be used to add more functions to theoptical fingerprint sensor module beyond the fingerprint sensing.

In the above examples where the fingerprint pattern is captured on theoptical sensor array via an imaging module, as in FIG. 4B and FIG. 6B,optical distortions tend to degrade the image sensing fidelity. Suchoptical distortions can be corrected in various ways. For example, aknown pattern can be used to generate an optical image at the opticalsensor array and the image coordinates in the know pattern can becorrelated to the generated optical image with distortions at theoptical sensor array for calibrating the imaging sensing signals outputby the optical sensor array for fingerprint sensing. The fingerprintsensing module calibrates the output coordinates referencing on theimage of the standard pattern.

In light of the disclosure in this patent document, variousimplementations can be made for the optical fingerprint sensor module asdisclosed. For example, a display panel can be constructed in which eachpixel emitting lights, and can be controlled individually; the displaypanel includes an at least partially transparent substrate; and a coversubstrate, which is substantially transparent. An optical fingerprintsensor module is placed under the display panel to sense the images formon the top of the display panel surface. The optical fingerprint sensormodule can be used to sense the images form from light emitting fromdisplay panel pixels. The optical fingerprint sensor module can includea transparent block with refractive index lower than the display panelsubstrate, and an imaging sensor block with an imaging sensor array andan optical imaging lens In some implementations, the low refractiveindex block has refractive index in the range of 1.35 to 1.46 or 1 to1.35.

For another example, a method can be provided for fingerprint sensing,where light emitting from a display panel is reflected off the coversubstrate, a finger placed on top of the cover substrate interacts withthe light to modulate the light reflection pattern by the fingerprint.An imaging sensing module under the display panel is used to sense thereflected light pattern image and reconstruct fingerprint image In oneimplementation, the emitting light from the display panel is modulatedin time domain, and the imaging sensor is synchronized with themodulation of the emitting pixels, where a demodulation process willreject most of the background light (light not from pixels beingtargeted).

As described above, display screens of portable electronic devices areoften implemented as an assembly of multiple layers. For example,display screens implemented as touchscreens can include display layersfor outputting video data, capacitive touchscreen layers for detectingtouch events, a hard top layer, etc. Additional layers are used tointegrate under-display optical sensing capabilities, such asfingerprint sensing. For light to reach the sensing components, thelight passes through the various layers between the top surface and thesensors (e.g., the photodetectors). To that end, the layers are designedto permit transmission of light, and some layers can be designed toenhance, bend, focus, collimate, reflect, and/or otherwise influencetransmission of light through the layers.

FIGS. 17A and 17B show an illustrative portable electronic device 1700,and a cross-section of an illustrative display module 1710 for such aportable electronic device 1700, respectively, according to variousembodiments. The portable electronic device 1700 is illustrated as asmart phone In other implementations, the portable electronic device1700 is a laptop computer, a tablet computer, a wearable device, or anyother suitable computational platform. The portable electronic device1700 can include a display system 423. As described above, the displaysystem 423 can be a touch sensing display system 423. The display system423 has, integrated therein, an under-display optical sensor. Asillustrated, the under-display optical sensor can define a sensingregion 615, within which optical sensing can be performed. For example,fingerprint scanning can be performed by the under-display opticalsensor when a user places a finger 445 on the display within the sensingregion 615. Such an under-display optical sensor can be implementedusing multiple layers.

The display module 1710 of FIG. 17B can be an implementation of thedisplay system 423 of FIG. 17A. As illustrated, the display module 1710includes a number of layers. A top cover layer 1715 (e.g., glass) canserve as a user interface surface for various user interfacingoperations. For example, the cover layer 1715 can facilitate touchsensing operations by the user, displaying images to the user, anoptical sensing interface to receive a finger for optical fingerprintsensing and other optical sensing operations, etc. In some embodiments,the display module 1710 includes the cover layer 1715 In otherimplementations, the cover layer 1715 is separate from the displaymodule 1710. For example, the display module 1710 is integrated into theportable electronic device 1700 as a module, and the cover layer 1715 isinstalled on top of the display module 1710.

One or more other layers of the display module 1710 form a liquidcrystal module (LCM) 1720. Below the LCM 1720, the display module 1710includes an enhancement layer 1725. As described herein, the enhancementlayer 1725 can include one or more layers of brightness-enhancementfilm, such as enhancement films including trapezoidal prism structures.The display module 1710 can further include some or all of a lightdiffuser 1730, a light guide plate 1735, a reflector film 1740, and aframe 1745. Some embodiments include additional components, such as oneor more display light sources 1750, and one or more external lightsources 1760 (e.g., for fingerprint and/or other optical sensing).

Implementations of the display light sources 1750 can include LCDdisplay backlighting light sources (e.g., LED lights) that provide whitebacklighting for the display module 1710. Implementations of the lightguide plate 1735 include a waveguide optically coupled with the displaylight sources 1750 to receive and guide the backlighting light.Implementations of the LCM 1720 include some or all of a layer of liquidcrystal (LC) cells. LCD electrodes, a transparent conductive ITO layer,an optical polarizer layer, a color filter layer, a touch sensing layer,etc. Implementations of the light diffuser 1730 include a backlightingdiffuser placed underneath the LCM 1720 and above the light guide plate1735 to spatially spread the backlighting light for illuminating the LCDdisplay pixels in the LCM 1720. Implementations of the reflector film1740 are placed underneath the light guide plate 1735 to recyclebacklighting light towards the LCM 1720 for improved light useefficiency and display brightness.

When the LCD cells in (e.g., in the sensing region 615) are turned on,the LCM 1720 (e.g., the LC cells, electrodes, transparent ITO,polarizer, color filter, touch sensing layer, etc.) can become partiallytransparent, although the micro structure may interfere and/or blocksome probe light energy. Embodiments of the light diffuser 1730, thelight guide plate 1735, the reflector film 1740, and the frame 1745 aretreated to hold the fingerprint sensor and provide a transparent orpartially transparent sensing light path, so that a portion of thereflected light from the top surface of the cover layer 1715 can reachsensing elements (e.g., a photo detector array) of the under-displayoptical sensor. The under-display optical sensor can include anysuitable components, such as fingerprint sensor pans, a photodetectorarray, an optical collimator array for collimating and directingreflected probe light to the photo detector array, and an optical sensorcircuit to receive and condition detector output signals from the photodetector array. Embodiments of the photodetector array include a CMOSsensor of CMOS sensing pixels, a CCD sensor array, or any other suitableoptical sensor array.

Embodiments of the enhancement layer 1725 include one or moreenhancement films. Some enhancement film designs include a prism filmwith sharp prism ridge and sharp prism valley profile (i.e., a sharptransition at each ridge, and a sharp transition at each Valley). Forexample, FIGS. 18A-18C show views of an illustrative portion of asymmetric enhancement layer 1800. FIG. 18A illustrates a zoomed-in view1810 of a small portion of the symmetric enhancement layer 1800 FIGS.18B and 18C show a cross-section of a small portion of one enhancementfilm layer 1820 of the symmetric enhancement layer 1800. FIG. 18C showsa cross-section of a small portion of two enhancement film layers 1820a, 1820 b of the symmetric enhancement layer 1800, stacked in orthogonalorientations with respect to each other.

As illustrated, each enhancement film layer 1820 is formed with a seriesof sharp prism structures. Each sharp prism structure includes a sharpridge 1822 and a sharp valley 1824. The zoomed-in view 1810 of FIG. 18Ashows the two enhancement film layers 1820 of FIG. 18C, stacked inorthogonal orientations with respect to each other, viewed from the top.As illustrated, the intersecting sharp prism structures form a grid ofsharp ridge lines 1812 and sharp valley lines ISM. correspondingrespectively to the sharp ridges 1822 and sharp valleys 1824 of eachsharp prism structure. As illustrated by FIG. 18C, the sharp ridges 1822point in the direction of the LCM 1720.

Such enhancement layers 1800 typically seek to enhance the brightness oflight directed toward a viewer, such as toward and/or through the LCM1720 For example, enhancement layers 1800 seek to enhance the brightnessof backlighting positioned behind the LCM 1720 and/or of probe lightingused for under-display optical sensing. As shown in FIG. 18B, lightpassing through the prism structures of the enhancement layer 1800 isbent in different directions, as illustrated by light paths 1832 a and1832 b. Each light path 1832 illustrates two directions of travel oflight energy. A first direction (i.e., generally toward the LCM 1720)can represent a vector of backlighting and/or probe lighting energy thatoriginated from a light source under the enhancement layers 1800. Asecond direction (i.e., generally away from the LCM 1720) can representa vector of backlighting and/or probe lighting energy that reflected offof another layer (e.g., the top transparent layer above the LCM 1720)and is traveling back in the direction of under-display optical sensors.

As light passes through the enhancement film layer 1820 in the directionof the LCM 1720, such bending can tend to be beneficial. For example,light passing through the enhancement film layer 1820 in the firstdirection, including light paths (e.g., paths 1832) having largeincident angles, can be bent in a generally convergent manner toward theLCM 1720, thereby causing brightness enhancement. As a corollary, lightpassing through the enhancement layers 1800 in the second direction cantend to be bent in a generally divergent manner. If optical sensing isattempted in that causes image blurring. In typical displayapplications, such blurring is of no concern, as the blurred light ispassing into the device and not toward the viewer. However, in contextof under-display optical fingerprint sensing, as described herein, suchblurring impacts light traveling in the direction of the optical sensingcomponents, which can frustrate optical sensing by components situatedbelow the conventional enhancement layer 1800.

For added clarity, FIG. I SB shows three example potential referencepositions (e.g., locations and orientations) 1850 for optical sensing,if an optical sensor is placed in accordance with reference position1850 a, the optical sensor can tend to detect light entering theenhancement film layer 1820 through the right and left prism faces,which can tend to cause image blurring However, if an optical sensor isplaced in accordance with reference position 1850 b or 1850 c. theoptical sensor can tend to detect light entering the enhancement filmlayer 1820 through only the right prism faces or the left prism faces,respectively. In such cases, blurring can be avoided, but at least halfof the detection area may not be imaged.

FIGS. 19A-19C show views of an illustrative portion of an asymmetricenhancement layer 1900, according to various embodiments. The asymmetricenhancement layer 1900 can be another embodiment of the enhancementlayer 1725. FIG. 19A illustrates a zoomed-in view 1910 of a smallportion of the asymmetric enhancement layer 1900. FIG. 19B shows across-section of a small portion of one enhancement film layer 1920 ofthe asymmetric enhancement layer 1900. FIG. 19C shows a cross-section ofa small portion of two asymmetric layers 1920 a, 1920 b of theasymmetric enhancement layer 1900, slacked in orthogonal orientationswith respect to each other.

As illustrated, each enhancement film layer 1920 is formed with a seriesof asymmetric prism structures. Each asymmetric prism structure(micro-prism structure) is generally defined by the cross-section havingtwo angled sides, forming a sharp ridge 1922 and a sharp valley 1924.Each of the two angled sides is slanted at a different respectivetilting angle 1926 relative to vertical, as illustrated. Notably, ateach extreme of the range of possible tilting angles 1926 is anembodiment in which one of the tilting angles 1926 is at substantiallyzero degrees, so as to effectively form a sawtooth-ridge prismstructure. In another embodiment, one tilting angle 1926 is 45 degrees,while the other is 52 degrees. In another embodiment, one tilting angle1926 is 45 degrees, while the other is 54 degrees In another embodiment,one tilting angle 1926 is 45 degrees, while the other is 56 degrees. Inanother embodiment, one tilting angle 1926 is 38 degrees, while theother is 52 degrees. In another embodiment, one tilting angle 1926 is 36degrees, while the other is 54 degrees. As described herein, the tiltingangles 1926 are selected to provide a desired type and/or amount ofbrightness enhancement (e.g., for backlight passing through theenhancement film layer 1920 in the direction of the LCM 1720).

The zoomed-in view 1910 of FIG. 19A shows the two enhancement filmlayers 1920 of FIG. 19C, stacked in orthogonal orientations with respectto each other, as viewed from the top. As illustrated, the intersectingprism structures form a grid of sharp ridge lines 1912 and sharp valleylines 1914, corresponding respectively to the sharp ridges 1922 andsharp valleys 1924 of each. Such an arrangement results in a top-downview that appears similar to that of the enhancement layer 1800 of FIG.18, but provides various features that are different from those of theconventional enhancement layer 1800.

FIG. 19B illustrates light traveling through the enhancement film layer1920 in the direction of the LCM 1720, for example, along light paths1930 Light generally passing through the enhancement film layer 1920 inthe direction of the LCM 1720 (i.e., having an upward directionalcomponent with reference to the illustrated orientation), such as thosefollowing light paths 1930 a and 1930 b are bent towards vertical by theangled surfaces of the micro-prism structures Thus, although certainlight paths are impacted differently by the asymmetric prism structuresthan by micro-prism structures of symmetric enhancement layer 1800, theasymmetric enhancement film layer 1920 still providesbacklight-enhancement features.

Unlike a symmetric enhancement layer 1800, the asymmetric enhancementfilm layer 1920 creates less blurring of light traveling in tiedirection opposite the LCM 1720 (i.e., having a downward directionalcomponent with reference to the illustrated orientation). FIG. 19B showslight traveling through the enhancement film layer 1920 in such adirection (e g., the direction of an under-display optical sensor), forexample, along light paths 1940. As illustrated, three objects 1950 arepositioned in different locations relative to the asymmetric enhancementfilm layer 1920. For example, the objects 1950 are fingerprint ridges orvalleys of a finger placed on the fingerprint sensing region of a devicehaving the asymmetric enhancement film layer 1920 disposed between anLCM 1720 and an under-display optical fingerprint sensor. Light from thesecond object 1950 b travels along refracted light path 1940 a todetection point “B” 1955 b (e.g., corresponding to a first potentialsensor location), while light from the third object 1950 c travels alongrefracted light path .940 b to detection point “C” 1955 c (e.g.,corresponding to a second potential sensor location). Notably, whileobjects 1950 b and 1950 c are relatively close together, theirrespective detection points 1955 b and 1955 c are relatively far apart.Light from the first object 1950 a travels along refracted light path1945 to detection point “A” 1955 a, after leaving the asymmetricenhancement film layer 1920 in a substantially vertical direction. Itcan be seen that configuring the sensor for detection of light exitingalong path 1945 (e.g., at detection location 1955 a) can yieldrelatively clear and bright detection information. This is furtherillustrated in FIG. 19C, in which the two stacked asymmetric enhancementfilm layers 1920 (in orthogonal orientations with respect to each other)can provide clear image light paths, such as represented by detectionpoint 1955 a.

Still, as discussed above with reference to FIGS. 18A-C, avoidance ofblurring can involve positioning the optical sensor to receive from onlyone of the left or right faces of the micro-prism structures. Forexample, a single optical sensor can be pointed to detect object 1950 aand object 1950 b via paths 1945 and 1940 a, respectively, as both lightpaths reach the sensor through right prism faces. Still, receivingoptical information from only one face of the micro-prism structures caneffectively reduce the sensing area that can be practically andaccurately imaged without blurting.

For the sake of illustration, FIG. 20 shows an illustrative fingerprintsensing region 615 overlaid on a single enhancement film layer 2090,such as enhancement film layer 1820 or 1920. As illustrated, themicro-prism structures of the enhancement film layer 2000 form parallelridge lines running generally in a first direction (labeled as an “X”axis 2020). and running generally orthogonal to a second direction(labeled as a “Y” axis 2025). Though not shown, it can be assumed that a“Z” axis points outward from the page, orthogonal to both the X and Yaxes, in the direction of the LCM 1720. As described above, though thetheoretical fingerprint sensing region 615 consumes a relatively largesensing area, an optical sensor oriented to sense the entire region cantend to receive reflected probe light from both prism faces of eachmicro-prism structure, thereby resulting in image blurring. An opticalsensor oriented to receive reflected probe light from only one of theprism faces of each micro-prism structure can tend to avoid imageblurring, but tends also to result in reducing the sensing region (asillustrated by sensing sub-region 2010). For example, the optical sensorcan be oriented to point generally in the negative-Y, positive-Zdirection (e.g., {0, -y, z}). In such cases, the practical sensingsub-region 2010 may be approximately half or less the area of thetheoretical sensing region 615.

FIG. 21 shows an illustrative fingerprint sensing region 615 overlaid onan enhancement panel 2100 having two enhancement film layers, stacked sothat their respective micro-prism structures run in generally orthogonaldirections. The enhancement panel 2100 can be implemented, for example,as enhancement panels shown in FIGS. 18A and 19A, or as the pair ofenhancement film layers shown in FIGS. 18C and 190. As illustrated, theenhancement panel 2100 has first micro-prism structures (e.g., of afirst enhancement film layer) forming parallel ridge lines runninggenerally in the first direction (along “X” axis 2020), and secondmicro-prism structures (e.g., of a second enhancement film layer)forming parallel ridge lines running generally in the first seconddirection (along “Y” axis 2025). With two enhancement film layers, eachlayer can cause blurring. For example, light passing through one prismface of a micro-structure of a lower enhancement film layer is acombination of light passing through both prism faces of amicro-structure of an upper enhancement film layer. As such, to avoidblurring, the optical sensor is oriented to receive reflected probelight from only one of the prism faces of each micro-prism structurethrough both enhancement film layers. For example, the optical sensorcan be oriented to point generally in the positive-X, negative-Y,positive-Z direction (e.g., {x, -y, z}). In such cases, the practicalsensing sub-region 2110 may be around one-fourth the area of thetheoretical sensing region 615. For example, the practical sensingsub-region 2110 may be less than one-fourth the area of the theoreticalsensing region 615 when using symmetric micro-prism structures (e.g.,like those illustrated in FIGS. 18A-18C), and the practical sensingsub-region 2110 may be a bit more than one-fourth the area of thetheoretical sensing region 615 when using well-designed asymmetricmicro-prism structures (e.g., like those illustrated in FIGS. 19A-19C).

Increasing the area of the theoretical sensing region 615 can counteractlimitations of a reduced-area practical sensing sub-region 2110. Forexample, doubling the area of the theoretical sensing region 615 candouble the area of the practical sensing sub-region 2110. One approachto increasing the area of the theoretical sensing region 615 is toincrease the distance between the object being detected (e.g., afingerprint feature) and a lens at the input of the optical sensorHowever, directly increasing such a distance can involve increasing thethickness of the optical sensor, which is impractical in manyapplications. For example, in smartphone applications, it is desirableto maintain a thin package, while still realizing a large sensing area.Embodiments seek to effectively increase the theoretical sensing region615 without a comparable increase in sensor thickness by usingrefraction and reflection to produce a folded light path.

FIG. 22 shows an illustrative under-display optical sensing environment2200, according to various embodiments. As illustrated, the environment2200 includes an under-display optical sensing system disposed below adisplay module 1710 The optical sensing system includes an opticalsensing module 2210, a refractive structure 2220, and a reflectivestructure 2230. The optical sensing module 2210 can include any suitablecomponents for optical sensing. For the sake of simplicity, the opticalsensing module 2210 is shown as an optical detector 2212 and a lens2214. For reference, the display module 1710 is illustrated as defininga display plane 2205. For example, a display surface of the displaymodule 1710 is generally flat (e.g., though it may include bezelededges, rounded edges, etc.), such that the display surface liesgenerally in the display plane 2205. The display module 1710 can be anysuitable type of display and can include one or more layers. In someembodiments, as described above, the display module 1710 is a liquidcrystal display (LCD) module having layers including a LCD layer (e.g.,including an array of LCD pixels, electrode interconnects, etc.).

The optical sensing module 2210 is configured to derive opticalinformation from received illumination energy. For example, as lightenters the optical sensing module 2210 through an input aperture 2216,the light is focused by the lens 2214 onto the optical detector 2212,which may include an array of photodetectors, and/or any other suitablecomponents. The received illumination energy can then be converted intooptical information. For example, the conversion can include anysuitable optical processing (e.g., using lenses, filters, modulators,masks, etc.) and/or any suitable logical processing (e.g., usingcomputational processors, state machines, software, etc.). The inputaperture 2216 can be oriented substantially parallel to the displayplane 2205, as illustrated. In such a configuration, a light pathtraveling parallel to the display plane 2205 can enter a center of theinput aperture 2216 on-axis with the lens 2214.

The reflective structure 2230 can receive reflected probe light passingthrough the display module 1710 (e.g., within an optical sensing region615) and can In some embodiments, the reflective structure 2230 includesa mirror integrated with (e.g., affixed to) a top reflective surface2235. For example, the reflective structure 2230 comprises structure tosupport and orient the mirror in an angle suitable for redirecting lightpaths 2240 of the reflected probe light toward the refractive structure2220. In other embodiments, the reflective structure 2230 is a unitaryprismatic body having an angled top surface 2235. In such embodiments,the unitary prismatic body can be made of a material (e.g., plastic,glass, etc. (having an index of refraction selected to reflect thereflected probe light that is incident on the angled top surface 2235after passing through the display module 1710 toward the refractivestructure 2220. As illustrated, the reflective structure 2230 is sizedand oriented to redirect light paths 2240 of the reflected probe lightfrom across the entire optical sensing region 615 toward the refractivestructure 2220. In certain implementations, the reflective structure2230 is sized and oriented to redirect light paths 2240 of the reflectedprobe light only from a portion of the optical sensing region 615,and/or additionally from areas outside the optical sensing region 615.

Embodiments of the refractive structure 2220 receive the reflected probelight from the reflective structure 2230 and bend the light paths 2240of the reflected probe light to converge on the input aperture 2216 ofthe optical sensing module 2210. For example, as illustrated, lightpaths 2240 a, 2240 b. and 2240 c originate from a leftmost edge, acenter, and a rightmost edge of the optical sensing region 615,respectively, thereby representing light paths 2240 from across theentire optical sensing region 615. Though covering a large opticalsensing region 615. the light paths 2240 are all redirected to therefractive structure 2220 by the reflective structure 2230, and are allhem by the refractive structure 2220 to converge on the input aperture2216 of the optical sensing module 2210. In some embodiments, therefractive structure 2220 is a unitary prismatic body having a firstrefractive surface (e.g., further from the optical sensing module 2210)to receive the reflected probe light from the reflective structure, anda second refractive surface (e.g., closer to the optical sensing module2210) to transmit the reflected probe light to the input aperture of theoptical sensing module, the first refractive surface being angled withrespect to the second refractive surface. For example, as illustrated,the second refractive surface can be oriented substantially orthogonallyto the display plane.

As described above (though not shown in FIG. 22), embodiments caninclude an illumination source and a top transparent layer. Theillumination source can be disposed under the display module 1710 togenerate probe light and to direct the probe light through at least aportion of the display module 1710. For example, the probe light sourcecan include a light emitting diode (LED), a vertical cavity surfaceemitting laser (VCSEL), or any other suitable light source. The toptransparent layer can be disposed above the display module 1710 andconfigured as an output interface for images produced by the displaymodule 1710. In some implementations, the display module 1710 alsoincludes touch-sensitive features, and the top transparent layer can befurther configured as an input interlace for touch-sensitiveinteractions. Though the optical sensing region 615 is illustrated aslaying substantially in the display plane 2205, the optical sensingregion 615 can also be defined with reference to a top surface of thetop transparent layer For example, when the optical sensing module 2210is configured for under-display optical fingerprint sensing, thefingerprint is typically placed on the top surface of the toptransparent layer, such that the optical sensing region 615 correspondsmore directly to the region of the top surface of the top transparentlayer within which optical fingerprint sensing can be performed. Theoptical sensing region 615, then, can be a region configured to receivethe probe light from the illumination source and to reflect a portion ofthe probe light responsive to interactions between an object (e.g.,fingerprint features) and the top surface. For example, as describedabove, the top transparent layer can he made of a material (e.g.,treated glass) having a particular index of refraction relative to theair above the top transparent layer and/or to a fingerprint feature orother object. As such, the probe light will tend to be reflected or notat the top surface of the top transparent layer based on whether or notan object is present (e.g., a fingerprint ridge will cause the probelight to be reflected, and a fingerprint valley will cause the probelight to be absorbed, scattered, etc.). In such embodiments, thereflected probe light being redirected by the reflective structure 2230,bent by the refractive structure 2220, and received by the opticalsensing module 2210 is the probe light being reflected by the topsurface of the top transparent layer.

In some embodiments, the reflective structure 2230 and the refractivestructure 2220 are configured (e.g., sized, shaped, positioned, andoriented) so that one of the light paths 2240 b passing through a centerof the optical sensing region 615 is redirected by the reflectivestructure 2230 and bent by the refractive structure 2220 so as to entera center of the input aperture 2216 of the optical sensing module. Forexample, the lens 2214 is a convex lens having a primary optical axis,and the input aperture 2216 is aligned with the lens 2214, so that acenter of the input aperture 2216 is aligned with the primary opticalaxis of the lens 2214. In some embodiments, as illustrated, thereflective structure 2230 and the refractive structure 2220 areconfigured so that one of the light paths 2240 b passing through thedisplay module in a first direction substantially normal to the displayplane (e.g., vertical with respect to the illustration) is redirected bythe reflective structure 2230 and bent by the refractive structure 2220so as to enter the input aperture 2216 of the optical sensing module2210 in a second direction substantially parallel to the display plane(e.g., horizontal with respect to the illustration).

FIG. 23 shows an illustrative under-display optical sensing environment2300 with off-axis sensing, according to various embodiments. Similar tothe environment 2200 of FIG. 22, the environment 2300 includes anunder-display optical sensing system disposed below a display module1710 The optical sensing system includes an optical sensing module 2210,a refractive structure 2220, and a reflective structure 2230 (with areflective top surface 2235. The optical sensing module 2210 can includeany suitable components for optical sensing, such as an optical detector2212 and a lens 2214. Unlike in FIG. 22, the environment 2300 of FIG. 23shows a practical optical sensing sub-region 2310 smaller than thetheoretical optical sensing region 615, such as described with referenceto FIGS. 20 and 21.

As described above, embodiments of the display module 1710 can includean enhancement layer having micro-prism structures For example, eachmicro-prism structures can have at least a first and second prism face,and the first and second prism faces can be symmetric or asymmetric withrespect to each other. As reflected probe light passed through themicro-prism structures, sonic of the reflected probe light passesthrough the first prism faces of the micro-prism structures, and otherof the reflected probe light passes through the second prism faces ofthe micro-prism structures. For example, a first portion of light paths2540 of the reflected probe light pass through first prism faces of themicro-prism structures, and a second portion of the light paths 2340 ofthe reflected probe light pass through second prism faces of themicro-prism structures, such that the first and second portions of thelight paths 2340 are divergent. As described above, such divergence cancause blurring if the optical sensing module 2210 receives both thefirst and second portions of the light paths 2340. Instead, embodimentscan configure the optical sensing module 2210 to derive the opticalinformation from the received illumination energy corresponding only tothe first portion of the light paths 2340 of the reflected probe light.As such, the practical optical sensing sub-region 2310 smaller than thetheoretical optical sensing region 615 (e.g., approximately one half orone quarter the area).

In some such embodiments, the enhancement layer includes a firstenhancement film layer and a second enhancement film layer. The firstenhancement film layer has a first portion of the micro-prism structuresarranged to form first parallel prism ridge lines running in a firstdirection, and the second enhancement film layer has a second portion ofthe micro-prism structures arranged to form second parallel prism ridgelines running in a second direction different from the first direction.For example, as illustrated in FIG. 21, the first enhancement film layercan be substantially identical to the second enhancement film layer, andthe first enhancement film layer can be slacked on top of die secondenhancement film layer and oriented so that the first direction isorthogonal to the second direction. In some such embodiments, the firstportion of the light paths 2340 of the reflected probe light passthrough the first prism faces of the micro-prism structures by passingthrough first prism faces of the first portion of micro-prism structuresand first prism faces of the second portion of micro-prism structures;and the second portion of the light paths 2340 of the reflected probelight pass through the second prism faces of the micro-prism structuresby passing through second prism faces of the first portion ofmicro-prism structures and second prism faces of the second portion ofmicro-prism structures There can also be a third portion of the lightpaths 2340 of the reflected probe light that pass through the firstprism faces of the first portion of micro-prism structures and thesecond prism faces of the second portion of micro-prism structures, anda fourth portion of the light paths 2340 of the reflected probe lightthat pass through the second prism faces of the first portion ofmicro-prism structures and the first prism faces of the second portionof micro-prism structures. Again, in such embodiments, the practicaloptical sensing sub-region 2310 can be smaller than the theoreticaloptical sensing region 615 (e.g., approximately only one quadrant of thetheoretical optical sensing region 615 may be usable without blurring).

Embodiments of the optical sensing module 2210 include the lens 2214 andthe optical detector 2212 (e.g., including any suitable type of sensingcomponents), oriented so that the reflected probe light caused toconverge on the input aperture 2216 of the optical sensing module 2210is focused onto the optical detector 2212 by the lens 2214. Some suchembodiments can be configured to prefer the first portion of the lightpaths 2340 that correspond to the illumination energy passing only (orprimarily) through a particular face of the micro-prism structures(e.g., only first prism faces). For example, as illustrated in FIG. 23,the under-display optical sensing system is configured so that lightpath 2340 a and light path 2340 b are directed into the optical sensingmodule 2210 y the reflective structure 2230 and the refractive structure2220, but light path 234 c is not. In some such embodiments, the opticaldetector 2212 can be positioned off-axis from the lens in accordancewith a generally off-center light path resulting from a configuredpreference only for the first portion of the light paths 2340. As such,the optical detector 2212 can be configured (e.g., positioned and/orsized) to receive illumination energy reflecting from the practicaloptical sensing sub-region 2310, and not from the rest of thetheoretical optical sensing region 615.

While this disclosure contains many specifics, these should not beconstrued as limitations on the scope of any invention or of what may beclaimed, but rather as descriptions of features that may be specific toparticular embodiments of particular inventions. Certain features thatare described in this patent document in the context of separateembodiments can also be implemented in combination in a singleembodiment. Conversely, various features that are described in thecontext of a single embodiment can also be implemented in multipleembodiments separately or in any suitable subcombination. Moreover,although features may be described above as acting in certaincombinations and even initially claimed as such, one or more featuresfrom a claimed combination can in some cases be excised from thecombination, and the claimed combination may be directed to asubcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. Moreover, the separation of various system components in theembodiments described in this patent document should not be understoodas requiring such separation in all embodiments.

Only a few implementations and examples are described and otherimplementations, enhancements and variations can be made based on whatis described and illustrated in this patent document.

A recitation of “a”, “an” or “the” is intended to mean “one or more”unless specifically indicated to the contrary. Ranges may be expressedherein as from “about” one specified value, and/or to “about” anotherspecified value. The term “about” is used herein to mean approximately,in the region of, roughly, or around. When the term “about” is used inconjunction with a numerical range, it modifies that range by extendingthe boundaries above and below the numerical values set forth. Ingeneral, the term “about” is used herein to modify a numerical valueabove and below the stated value by a variance of 10%. When such a rangeis expressed, another embodiment includes from the one specific valueand/or to the other specified value. Similarly, when values areexpressed as approximations, by use of the antecedent “about,” it willbe understood that the specified value forms another embodiment. It willbe further understood that the endpoints of each of the ranges areincluded with the range

All patents, patent applications, publications, and descriptionsmentioned here are incorporated by reference in their entirety for allpurposes. None is admitted to be prior art.

What is claimed is:
 1. An electronic device comprising: a display modulethat defines a display plane, and an under-display optical sensingsystem positioned below the display module and comprising: an opticalsensing module having an input aperture oriented substantially parallelto the display plane, the optical sensing module to derive opticalinformation from received illumination energy; a refractive structure;and a reflective structure to receive reflected probe light passingthrough the display module and to redirect light paths of the reflectedprobe light toward the refractive surface, wherein the refractivestructure is to receive reflected probe light from the reflectivestructure and to bend the light paths of the reflected probe light toconverge on the input aperture of the optical sensing module.
 2. Theelectronic device of claim 1, wherein the refractive structure and thereflective structure are configured so that one of the light pathspassing through the display module in a first direction substantiallynormal to the display plane is redirected by the reflective structureand bent by the refractive structure so as to enter the input apertureof the optical sensing module in a second direction substantiallyparallel to the display plane.
 3. The electronic device of claim 1,wherein the refractive structure is a unitary prismatic body having afirst refractive surface to receive the reflected probe light from thereflective structure, and a second refractive surface to transmit thereflected probe light to the input aperture of the optical sensingmodule, the first refractive surface being angled with respect to thesecond refractive surface.
 4. The electronic device of claim 3, whereinthe second refractive surface is oriented substantially orthogonally tothe display plane.
 5. The electronic device of claim 1, wherein thereflective structure is a unitary prismatic body having an angled topsurface, the unitary prismatic body having an index of refractionselected to reflect the reflected probe light that is incident on theangled top surface after passing through the display module.
 6. Theelectronic device of claim 1, wherein the reflective structure comprisesa mirror.
 7. The electronic device of claim 1, further comprising: anillumination source disposed under die display module to generate probelight and to direct the probe light through at least a portion of thedisplay module; and a top transparent layer disposed above the displaymodule and configured as an output interface for images produced by thedisplay module and having an optical sensing region configured toreceive the probe light from the illumination source, the toptransparent layer configured so that a portion of the probe light isreflected by a top surface of the top transparent layer responsive tointeractions between an object and the top surface, the reflected probelight being the portion of the probe light that is reflected by the topsurface.
 8. The electronic device of claim 7, wherein the refractivestructure and the reflective structure are configured so that one of thelight paths passing through a center of the optical sensing region isredirected by the reflective structure and bent by the refractivestructure so as to enter a center of the input aperture of the opticalsensing module.
 9. The electronic device of claim 1, wherein the opticalsensing module comprises a lens and an optical sensor, oriented so thatthe reflected probe light caused to converge on the input aperture ofthe optical sensing module is focused onto the optical sensor by thelens.
 10. The electronic device of claim 1, wherein: the display modulefurther comprises an enhancement layer having a plurality of micro-prismstructures; a first portion of the light paths of the reflected probelight pass through first prism faces of the plurality of micro-prismstructures; a second portion of the light paths of the reflected probelight pass through second prism faces of the plurality of micro-prismstructure, such that the first and second portions of the tight pathsare divergent; and the optical sensing module is configured to derivethe optical information from the received illumination energycorresponding only to the first portion of the light paths of thereflected probe light.
 11. The electronic device of claim 10, wherein:the enhancement layer includes a first enhancement film layer and asecond enhancement film layer, the first enhancement film layer having afirst plurality of the micro-prism structures arranged to form a firstplurality of parallel prism ridge lines running in a first direction,the second enhancement film layer having a second plurality of themicro-prism structures arranged to form a second plurality of parallelprism ridge lines running in a second direction different from the firstdirection; the first portion of the light paths of the reflected probelight pass through the first prism faces of the plurality of micro-prismstructures by passing through first prism faces of the first pluralityof micro-prism structures and first prism faces of the second pluralityof micro-prism structures; the second portion of the light paths of thereflected probe light pass through the second prism faces of theplurality of micro-prism structures by passing through second prismfaces of the first plurality of micro-prism structures and second prismfaces of the second plurality of micro-prism structures; a third portionof the light paths of the reflected probe light pass through the firstprism faces of the first plurality of micro-prism structures and thesecond prism faces of the second plurality of micro-prism structures,and a fourth portion of the light paths of the reflected probe lightpass through the second prism faces of the first plurality ofmicro-prism structures and the first prism faces of the second pluralityof micro-prism structures.
 12. The electronic device of claim 11,wherein: the first enhancement film layer is substantially identical tothe second enhancement film layer; and the first enhancement film layeris stacked on top of the second enhancement film layer and oriented sothat the first direction is orthogonal to the second direction.
 13. Theelectronic device of claim 10, wherein: the optical sensing modulecomprises a lens and an optical sensor, oriented so that the reflectedprobe light caused to converge on the input aperture of the opticalsensing module is focused onto the optical sensor by the lens; the firstportion of the light paths correspond to an off-center light pathrelative to the lens; and the optical sensor is positioned off-axis fromthe lens in accordance with the off-center light path to receiveillumination energy corresponding primarily to the first portion of thelight paths.
 14. The electronic device of claim 10, wherein theplurality of micro-prism structures comprise symmetric micro-prismstructures.
 15. The electronic device of claim 10, wherein the pluralityof micro-prism structures comprise asymmetric micro-prism structures.16. The electronic device of claim 1, wherein the display module is aliquid crystal display (LCD) module having a plurality of layersincluding a LCD layer..
 17. The electronic device of claim 1, whereinthe electronic device is a smartphone. 18 An under-display opticalsensing system for installation under a display module that defines adisplay plane, the system comprising: an optical sensing module havingan input aperture oriented to receive illumination energy, the opticalsensing module to derive optical information from received illuminationenergy; a refractive structure; and a reflective structure, thereflective structure and the refractive structure arranged with respectto each other and to the optical sensing module, such that light pathsof reflected probe light passing through an optical sensing region ofthe display module is redirected by the reflective structure toward therefractive surface, and bent by the refractive structure to converge onthe input aperture of the optical sensing module.
 19. The system ofclaim 18, wherein: the optical sensing module comprises a lens and anoptical sensor, oriented so that the reflected probe light caused toconverge on the input aperture of the optical sensing module is focusedonto the optical sensor by the lens.
 20. The system of claim 19,wherein: the optical sensing module is to derive optical informationfrom received illumination energy corresponding to a portion of thelight paths of the reflected probe light passing through a particularsub-region of the optical sensing region; and the optical sensor ispositioned off-axis from the lens to receive the illumination energy viaan off-center light path relative to the lens.
 21. The system of claim20, wherein: the particular sub-region corresponds approximately to onequadrant of the optical sensing region.